5 Overlooked challenges of using LLMs in organisational decision-making

Large language models (LLMs) are a type of generative artificial intelligence (GenAI) designed to understand and generate human language.

The increasing use of LLMs in organisations — through tools like Copilot, for example — calls for a careful reflection on their relationship with knowledge management (KM), specially in the context of organisational decision-making.

In this text, I identify five challenges that bring me great cause for concern and which I fail to properly find reflected in the thousands of articles dedicated to the topic of GenAI and LLMs.

???? No KMOL pode encontrar uma vers?o deste texto em língua portuguesa.

LLMs bring with them the promise of faster and easier access to the “right” knowledge, thus increasing the efficiency and effectiveness of organisational decision-making.

It is no surprise, then, that organisations start to experiment with LLMs: to transcribe meetings and write summaries, aid in content creation, act as brainstorming companions, assist with complex queries, support onboarding and training, perform data analysis and many others.

While LLMs represent an exciting opportunity for knowledge management, some challenges have been identified:

• Data security and privacy concerns: Sensitive corporate data might be leaked, intellectual property may be compromised, and data protection regulations might be breached. This might happen if the LLMs aren’t properly secured, but, from my experience, it is more likely to happen due to insufficient training and oversight when it comes to define and manage access permissions.
• Accuracy and reliability: LLMs may generate plausible-sounding but incorrect or misleading information. Who has never heard stories of LLMs hallucinations and incorrect responses?
• Knowledge silos: LLMs can perpetuate existing silos if they’re not trained on a comprehensive dataset. The limits of the dataset can be indadvertedly set by a poorly defined and maintained set of access permissions.
• Resistance to adoption: Employees may distrust LLMs, may struggle to incorporate them into their workflows.
• Automation bias: Employees may rely too much on the LLM’s outputs, using them without applying the necessary human expertise and judgment (or, even despite applying their judgement – i.e., trusting the machine more than they trust themselves).

These well-documented challenges, however, only scratch the surface. The intersection of LLMs with knowledge management reveals more nuanced concerns.

Making decisions on incomplete information

One of AI's key promises is enhancing decision-making quality. Ironically, this promise may be compromised by LLMs’ illusion of comprehensive insight.

Because LLMs respect data access permissions, employees may receive answers based only on data they can access, unaware of critical information elsewhere in the organisation.

Combined with AI's authoritative tone, this reduces the likelihood of employees seeking clarification from colleagues - a crucial part of collaborative validation and critical thinking.

A potential approach to mitigate this risk involves providing transparency and context within the LLM's responses.

For example, the system could clarify: “Based on the information you have access to, this is the response. However, additional information may exist within the company that could influence this outcome. We recommend consulting with [names of colleagues the LLMs identifies as subject matter experts].”

This way, the model respects data permissions, encourages critical assessment, and identifies colleagues who can be of assistance. (The latter, by the way, is a critical element for improving knowledge flow, thus increasing organisational learning.)

External data sources

Building on the situation above, considering corporate LLMs are mostly being trained on internal data, what if an important part of the answer comes from information only available outside the organisation? Will the answer be based solely on internal information, possibly out-of-date and most likely incomplete?

LLMs for experts; SMLs for beginners

Paradoxically, LLMs are likely to generate better results when used by subject matter experts. The pre-existing knowledge is crucial to write optimal questions (prompts), check for errors and inconsistencies, identify risks associated with acting on the information provided, fine-tune and “debug” responses, etc.

While AI can be used for onboarding and training, small language models (SMLs) are probably a better approach for that use case, as they can be specifically trained on a controlled dataset tailored to the organisation’s needs and suited to the users' expected level of knowledge. Unlike LLMs, SMLs do not rely on broad, generalised datasets, reducing the risk of misinformation and the need for pre-existing knowledge from the user.

Out-of-date

Internal data often lacks proper maintenance.

Many organisations do not think to define policies, processes and workflows to ensure that information stored in their internal systems is and remains up-to-date and correct. Some, at least associate a “review by date” to their stored information. This metadata can be used as an indication of reliability.

I am not certain if current LLMs allow for the exclusion of training data based on metadata fields, but I’d venture to say that, even if they do, few organisations are likely to be making such configurations.

This means outdated information could influence LLM responses, leading to flawed decisions - especially given users' tendency to trust AI's authoritative outputs.

It is critical to have a solid governance model which covers the entire lifecycle of data, information and documented knowledge – including its review and archival. (More on governance below)

Access

The risk of employees accessing information they shouldn't due to improperly configured permissions, is likely as high as the risk of them being unable to find the information they need, because permissions are too restrictive.

These exaggerated restrictions often stem from the traditional mindset where information is seen as a source of power, leading organisations to adopt a 'restrict by default' approach. In such environments, access is frequently granted on a case-by-case basis, only when a strong justification is provided, which can hinder effective knowledge sharing.

To address this, organisations should balance data security with accessibility by aligning access controls with their broader knowledge-sharing goals. Employees should be able to access the information necessary for their roles without compromising security.

Achieving this requires a governance model that is both conscientious and adaptable—one that ensures permissions are carefully maintained, reviewed regularly, and clearly and periodically communicated across the organisation.

Conclusion

I am far from opposing the use of LLMs. In fact, I used ChatGPT and Claude for drafting this post. However, their current limitations make me cautious and prevent me from encouraging its wide corporate use for decision-making.

Without proper systems and processes in place, LLMs risk repeating the Knowledge Management systems' fate from the early 2000s - where hasty implementation without adequate support led to disappointing results and damaged KM's credibility. The technology's potential is clear, but successful implementation requires careful consideration of the challenges outlined above.