The temporary definitive guide to building and operating LLM solutions in production environments

I would love it if Cohere and OpenAI would write documentation on this subject. I call this "temporary" because everything is changing so fast.

Why am I asking for these docs? Because it will open up the market opportunity for their products. (more on this below)

This is a bit of a rant and a callout to the recent $10M seed round LangChain just closed.

When you read the documentation at Cohere and OpenAI , the best docs for using commercially available LLM's today, you might arrive at the thought that, hey they have all of the stuff I need. If you are implementing the most simple of GenAI solutions, many of which are very powerful and not to be discounted, then yes they do have what you need, and that's cool.

However, once you start bringing your own data to the party and start thinking about things like automating customer service and support, or you want to fine-tune a model (or even wonder if you should) you are in a whole new ball of wax that gets complicated fast.

What does it take to get a solution like this into a production that is safe, performant, cost-effective and dependable over time? What third-party tools are you going to need? Who are the players? Is there a sample code that shows how a more complex LLM solution with multiple applications is strung together? How do I test, version control and monitor the system over time? When should I use vector/semantic/neural search over fine-tuning? What is the temporary definitive guide to fine-tuning?

This is where companies like LangChain, Humanloop , Vectara , Pinecone , Weights & Biases Robust Intelligence come in. There are many more...

I call this document temporary because best practices are often not yet known and/or are evolving rapidly. I've watched videos where experts at 美国斯坦福大学 and places like Humanloop are in wonder and amazement at how fast this is all changing.

If I were at Cohere , OpenAI , Anthropic , AI21 Labs ... I would have sufficient effort in place to create this documentation and keep it updated as part of our formal help and learning center. This documentation would be as important as the product itself. This requires a dedicated durable team.

Why? Because it will open up the market opportunity for their products. How big is the current market for LLMs? Well, given LLMs can bring value to anyone who uses a computer / smart device, the market is MASSIVE. Companies are rushing to figure out how to add AI to their products. Yet to build something you need a RARE set of skills. This means we need to educate ourselves to expand the near-term and future market opportunity.

Check out this article about LangChain. They are building a lot of the needed connective tissue.

Is today Wednesday?