Lesson learnt from the DPD AI Chatbot swearing blunder

On 19th January 2024, BBC news reported that a courier company DPD had to partially shut down its AI Chatbot service (DPD error caused chatbot to swear at customer - BBC News) after it swore at a customer. This incident highlights the challenges companies face when deploying AI Chatbots without proper governance in place. As the news article has pointed out that AI Chatbots can often be manipulated into generating responses they weren’t designed to give. But what causes this vulnerability?

In the past slightly over a year since the emergence of Generative Pre-Trained Transformer (GPT) technology to the consumer market, many AI Chatbots are built using the large language model (LLM) based on the GPT which has over 175 billion parameters. The core architecture of these powerful and formidable models is designed to produce fluent, contextual text based on whatever prompts and commands they receive as inputs. Unfortunately, unlike humans, the AI Chatbots do not have their own intentions, judgement or ability to intentionally disobey instructions. The nature of GPT is that it has an objective to follow prompts and generate relevant and cogent responses without any discernment as to whether the content is appropriate or not. In other words, the AI doesn’t have the “common sense” like humans.

In the DPD’s case, it seems that it is quite likely the company subscribed / licensed the pre-trained model, connected it to its internal corporate data, but did not sufficiently configure, customise or even constrain the model’s innate and obedient generativity. I can speculate that as a result when customers submitted innocuous requests (eg: write a poem) or deliberately submitted problematic requests (eg: write an offensive poem with derogatory content), the AI Chatbot automatically did as commanded. It lacked filters or inhibitors to stop it from following inappropriate instructions.

Well, whoever is in-charge with the AI Chatbots in DPD has obviously failed to implement adequate AI governance before deploying this capability to customers. They should have configured the system appropriately and tested rigorously to ensure unacceptable and inappropriate requests are filtered out. The AI model needs to be re-trained thoroughly before deployment. Rigorous testing scenarios should have been run to ensure the AI Chatbot could safely handle malicious inputs. The AI model probably needs further fine-tuning on appropriate data and guardrails incorporated to prevent unacceptable text generation under any circumstances.

This public blunder serves a wakeup call to organisations excitedly racing to deploy AI Chatbots. Such powerful technologies must be appropriately governed just like any other COTS (commercial off the shelf) and enterprise software products. The AI models need thoughtful design adaptions, ongoing monitoring and ethical oversight to align them with organisational policies, societal values, regulations and cybersecurity. Organisations should not “naively” plug-and-play pre-built and pre-trained AI models provided by external vendors, and expect them to behave properly without governance being embedded into their design processes. Shortcuts and lack of considerations often lead to preventable harms.