How Vector Databases Help Avoid Expensive, Eloquent, Wrong GenAI Answers (Vol. 10)

GenAI is expensive. Its answers can also be wrong. This week, we explore how vector databases can help.

Training your own LLMs is expensive

“There's a common narrative that running your model is too expensive and much cheaper to use an API to run large language models. However, anyone who's used GPT-4-32K at scale will tell you that you can easily spend the cost of buying an A100 GPU.”

Using GenAI at scale can run up eye-popping charges quickly. Michael Bommarito , a tech entrepreneur and advisor, described spending $13,000 to train GenAI for his single project —about as much as buying a data-center-grade NVIDIA A100 graphical processing unit (GPU).

To learn about Michael’s test, read his post on LinkedIn.

LLMs can easily give confident, elegant—and wrong—answers

For example, Andreessen Horowitz explains that asking an LLM for Apple’s gross margin last quarter can easily yield a confident, incorrect answer of $63 billion because:

1. LLMs are so expensive to train that they have months-or years-old data.
2. With a company name like “Apple,” the LLM could logically choose revenue and cost numbers from a random fruit company’s financial statements.
3. Its gross margin calculation is not mathematically correct.

This happens because LLMs aren’t designed to solve these kinds of questions. They’re prediction machines, not math machines. They’re trained on vast amounts of third-party internet data. Often, the data you need isn’t in the training set, like financial results. Vector databases help get less expensive and correct answers.

Read The key to unlocking the power of generative AI by Noel Yuhanna from Forrester in Future CIO.

Vector databases can help reduce GenAI’s costs and provide more accurate, fresh results

A16Z continues to explain how vector databases help GenAI apps reduce LLM training costs and provide more accurate results in three ways:

1. They reduce LLM API calls: Vector databases help optimize cost by reducing API calls with smart data preprocessing, breaking documents into smaller chunks, generating numerical encodings, or embeddings, of that data, and using them for queries. This reduces the context needed for answers. It’s much less expensive to store the correct answer than all the documents required to “guess” at your cost of goods.
2. They provide mathematically correct answers: By using vector embedding to compute the mathematical parts of GenAI’s answers and then combining correct data with language context, you can get eloquent answers that are factually correct.
3. They can provide cost-effective fresh data: Vector databases combat staleness by complementing language context with real-time data. Answers stay fresh without the retraining cost.

Read GenAI's Act II and Vector Databases to explore the emerging enterprise applications using GenAI and how vector databases fit.

Subscribe to {The, Weekly, Vector ↗?}

It’s an exciting time to join the GenAI and vector database community. To keep track of it all, subscribe to {The, Weekly, Vector ↗?}.