LLM Agnosticism, RAG over Fine-tuning, SharePoint Integration – lessons from the frontline

I’ve been averaging about five customer calls a week since December. As CEO, these discussions are critical. They help VisibleThread (VT) stay 100% aligned with the business objectives of the enterprise accounts we work with.

To remind, we work with customers in regulated sectors like defense & space, healthcare, IT services, government agencies etc. So, security is a top priority for these organizations and their IT colleagues. Especially when considering the adoption of any solution that employs AI or Generative AI (GenAI).

To be clear, many product companies handling non-sensitive info don’t need to worry as much about security as we do. For example, if you’re building a product in the MarTech (Marketing Technology) space, then it’s fine to use the public cloud.

But for our enterprise customers, the public cloud is simply a non-runner. It’s on-premises, private cloud (often MS Azure GCC High) OR bust! Zero data beyond the firewall.

So, this week I want to share three themes I’m hearing over and over when chatting with customers. If you’re evaluating products that use GenAI, you might find it helpful.

Quick Context Set on VT Writer - where we leverage GenAI

We built VT Writer to be an AI-powered writing aid designed for organizations who create and review sensitive content. It is a browser-based software application (with an optional MS Word Addin) that creates 1st draft content grounded in your proprietary content, scores content for clarity / accessibility and provides suggestions for how you can improve content quality. It can run 100% securely, with no data going outside your network.

Theme 1 – Vendor should be LLM Agnostic

VT Writer is LLM-agnostic. This means it can integrate with any major LLM (Large Language Model) for the tasks that require GenAI. We were always very intentional about this decision, figuring enterprises would want choice about the LLM they want to use. Turns out we were right.

So, why is LLM-agnosticism so important?

• Customers are standardizing on their GenAI stack, and specifically the LLMs that they standup and support. Back in the day, we used to joke “You never get fired for using IBM”. The modern version of this is “you never get fired for using an LLM from Microsoft, eg: GPT4 in Azure”.
• For this reason, Apps like VT Writer etc. need to integrate with the approved LLM supported by IT. The word “integrate” is key.
• When we tell our customers that “we integrate, we don’t ship” the LLM, you can almost hear the collective sigh of relief from IT. A potential deal breaker removed.
• The other big benefit for the “integration” approach is that if a new groovy model comes along, then our customers can use it with impunity. No adverse impact on VT Writer. Plug and play, the app will continue working and take advantage of hooking into the new model. Again, this keeps everyone happy.

Takeaway: If you’re evaluating app vendors that leverage GenAI, make sure they integrate with your standard LLMs. IT are overloaded, they want to hear that you “get it”. Have the vendor talk to your IT team early in your evaluation. Once they hear that the vendor you’re evaluating supports Ollama* for example as a way to support plug and play, they‘ll get massive comfort. It dramatically lowers risk all round.

* Ollama is an abstraction layer over the LLM. It allows App developers swap out LLMs simply and easily.

Theme 2 – Tapping into proprietary data, why RAG wins, fine tuning loses

There are two ways to leverage your proprietary data when using GenAI solutions:

1. Fine-tune an LLM with a proprietary data set. This means you have a new model to deploy OR
2. Use Retrieval Augmented Generation (RAG) layered with a general LLM.

NOTE: There’s a time and a place for fine-tuning. For highly specific domains, for instance gene-editing, then highly specialized, fine-tuned LLMs, or Small Language Models (SLMs) make a ton of sense. But for more general productivity use cases like 1st draft content creation, text rephrasing or simplification etc., any of the general-purpose leading models like GPT, Llama, Mistral & many others work just great. And using RAG to combine the prompt with proprietary data contained in SharePoint files or elsewhere works perfectly well.

So, why does RAG win over fine-tuning?

• RAG means if your underlying proprietary data changes or updates, you automatically get those updates.
• RAG means that you can attribute the results from the LLM to specific sources in your proprietary data. So, for example you’ll see from the screen below, that our response to the question “how does VisibleThread handle security?” is grounded with references to specific source content.
• RAG reduces the risk of hallucinations, since it grounds the answers in your proprietary data.

Why fine-tuning is problematic?

• If you have a fine-tuned model, where the fine-tuning was based on your proprietary content, once the fine-tuning step completes, then the model is static. In other words, it will not reflect any subsequent updates to your reference data. This makes the fine-tuned model stale, after updated or new content. And it’s an expensive and onerous task to expect organizations to constantly fine-tune models.
• Security and permissions. If you fine-tune a model based on a corpus of documents and content, how do you know that certain users querying the fine-tuned LLM will not receive answers based on content that they have no right to see? This is a big issue from a data privacy and permissions standpoint.

Takeaway: If you are evaluating productivity solution vendors, and they explain they use RAG to tap into proprietary data, that’s good. But if the vendor suggests they’ll fine-tune based on your corpus of data, then be a little more circumspect. It’s likely going to be a problem for the reasons I mentioned.

Theme 3 – For proprietary data, support SharePoint integration

For the customers we work with, SharePoint is the dominant platform for storing documents. Typically, these are in SharePoint sites, often a site reflects a strategic customer or possibly a strategic pursuit. How customers organize data is of course dependent on their business needs etc.

Now, assuming the vendor supports an LLM + RAG approach to ground the answers from GenAI, then the next thing I’m hearing in pretty much every discussion is the need to support SharePoint.

So, why is SharePoint Integration so important?

• Sounds obvious, but if your data is in SharePoint, you don’t want the extra friction of downloading it to a hard drive or network drive. Instead, you just want to point at the specific SharePoint files.
• You also want to automatically get any updated versions from SharePoint.
• Finally, permissions and security. This is very important, the vendor really needs to have clear permissions policies in place for SharePoint access. They need to use SharePoint delegated permissions so everything is above board. Fortunately, VT Writer does all this, so we’re good to go.

In summary, what I hear practically on every call are the above three themes. If you’re exploring GenAI products, it’s likely they’ll be part of the conversation for you too.?

Till next time.

---?

Best,

Fergal

Founder & CEO of VisibleThread

PS: Please subscribe to The Human Factor biweekly newsletter where I share more insights like this.