Understanding LLMs as a Product Manager | Part 1 of 3

Large Language Models (LLMs) are everywhere. For us PMs, it's not enough to just know that LLMs are “powerful” or “transformational.” What truly matters is understanding how they work—because that’s what enables us to make better decisions when building products.

Having worked extensively on AI observability, I’ve had first-hand exposure to the inner workings of LLMs—how they process data, generate insights, and where they can go wrong. Through this series, I’ll break down the complexities of LLMs for the product community—without the jargon overload. ??

Lets go!

Tokens: The Raw Ingredients of an LLM

Before an LLM can process text/image/Audio/Video etc, it first chops it up into smaller pieces—called tokens. These aren’t always full words; they can be subwords, characters, or even punctuation, for non-text inputs (like images or audio), tokens can be pixel values or encoded waveforms.

• Example: “Artificial Intelligence” → [‘Artificial’, ‘Intelli’, ‘gence’]
• Most LLMs work with 32,000 to 50,000 tokens, allowing them to process text efficiently.

?? Why Should You Care?

Every time you interact with an LLM, you’re charged per token—so tokenization affects cost, speed, and response quality. More tokens = better nuance but also higher costs.

Embedding Vectors: Turning Words into Numbers

Once a sentence is tokenized, each token is transformed into an embedding vector—a fancy way of saying: A mathematical representation that captures the meaning and relationship between words. Think of it like a map, where similar words are closer together. Each embedding has hundreds or even thousands of dimensions, depending on the model. The idea is that embedding dimensions capture different linguistic and semantic properties of a token.

?? After this step, LLMs never work with raw words again—just these embedding vectors.

?? Why Should You Care?

• More embedding dimensions = better understanding of nuance but also higher compute costs.
• If you’re building a simple AI assistant, GPT-3.5 might be enough. But if you’re handling contract analysis or financial compliance, GPT-4 or Claude 3 is worth the extra cost.

The Transformer Engine: Making Sense of Context

Now that everything’s converted into vectors, LLMs use transformers to process the information.

?? Step 1: Self-Attention Mechanism – The model looks at each word in relation to every other word in the sentence. Example: In "She unlocked the vault with the key," the model figures out that "key" refers to "vault", not "unlocked".

?? Step 2: Feedforward Layers – These refine the meaning further, filtering out noise and strengthening relevant connections.

?? Why Should You Care?

Transformers are why LLMs don’t just predict words randomly—they deeply understand the context. In transformers, attention is computed in each layer. Having more layers can help capture deeper contextual relationships, but the quality of attention comes from the design and implementation rather than just the layer count. While GPT-3 is a deep transformer model, its largest version is typically noted to have 96 layers. So depending on the use case, select the model to ensure you value cost vs attention context

The Real-World Trade-Offs for Product Managers

Now that we know the core mechanics, let’s talk about why this actually matters when building AI products:

?? Scalability vs. Cost:

• Bigger models (GPT-4, Claude, Gemini 1.5) = better reasoning, but higher inference cost.
• Smaller models (GPT-3.5, Mistral, Llama-2) = cheaper, but may miss nuance in responses.
• Choosing the right model is about balancing cost vs. accuracy for your use case.

?? Accuracy vs. Relevance:

• More embedding dimensions = better precision, but also higher processing costs.
• If you're running a simple FAQ bot, a lightweight model is fine.
• If you need deep legal or medical insights, you need more embeddings & a stronger transformer architecture.

?? Deep Understanding of Use Cases Matters:

• The real key to choosing the right model isn’t just about size or embeddings—it’s about what your customers need. It's easy to just pick the "biggest" model—but often, smaller models fine-tuned for a specific task perform just as well. The key is understanding where accuracy matters, where cost trade-offs exist, and what your users need.

?? What’s Next?

This is Part 1 of my 3-part series on "Understanding LLMs as a Product Manager"—next, I’ll break down Fine-Tuning vs. RAG: Which One Should You Use?

?? What’s the biggest challenge you’ve faced while working with LLMs? Let’s discuss this in the comments! ??

?? Credits & Inspiration

This post is inspired by my research, customer discussions, and insights from the AI community. A special mention to the 3Blue1Brown, whose incredible visual explanations of transformers and embeddings helped shape my understanding.