The Business Case for Small AI Models: Efficiency Without Compromise

In the rapidly evolving landscape of artificial intelligence, business leaders face a critical decision: which AI models will deliver the most value to their organization? While much attention has focused on massive models with billions of parameters, there's a compelling case for considering smaller, more specialized AI models. Lopez Research interviewed Kate Soule, Director of Technical Product Management for IBM's Granite products, to learn more about these smaller models.

IBM Granite represents a family of open AI models specifically designed for transparency, data governance, and practical business applications. IBM's third-generation of Granite models (3.2) offer reasoning capabilities and multimodal models that offers vision and has been optimized for document understanding.

Why Size Matters (and Smaller Can Be Better)

Soule says, "Everything gets more difficult as the model gets larger." This challenge manifests in several business-critical ways, such as:

• Higher operational costs. Larger models require more computing resources and energy, directly increasing operational expenses.
• Increased latency. Bigger models take longer to generate responses, potentially degrading customer experience.
• Need for powerful hardware. If using larger models in the cloud or on-premises, these models demand more powerful and expensive GPU infrastructure.
• Limited customization. Adapting massive models to your business needs requires substantial computing resources and expertise.

The Efficiency Advantage

IBM's approach with their Granite models focuses on efficiency through purpose-built smaller models. At just 2 to 8 billion parameters (compared to the largest industry models, such as Llama 3.1 from Meta, exceeding 400 billion parameters), these models deliver several key advantages, including:

1. Cost-effectiveness. Lower computational requirements translate directly to reduced operational costs.
2. Customization potential. Smaller models are more manageable and less expensive to fine-tune for specific business tasks.
3. Deployment flexibility. Some models are small enough to run locally on standard hardware, such as an AI PC, eliminating cloud dependency.
4. Faster response times. Reduced model size means quicker inference and better user experiences.

The "Fit-for-Purpose" Approach

Rather than seeking a single "model to rule them all," business and IT leaders are working together to assemble a portfolio of AI solutions that incorporate foundational LLMs with specialized AI models to support different business needs. As Soule notes, "To get value and to be able to deploy AI cost-effectively... you need to consider having fit-for-purpose models."

The fit-for-purpose approach allows businesses to optimize performance and cost based on the specific requirements of each use case. A larger model may be the best solution for high-value, complex tasks. For routine operations, a smaller specialized model often delivers comparable results at a fraction of the cost.

Soule shared that IBM's Granite models embody this philosophy with their modular design. Instead of trying to create a single massive model for all tasks, IBM has developed distinct models for different enterprise functions. For example, IBM’s Granite solutions offer the models for use cases such as coding, time-series forecasting, security and language models for designed for agentic workflows, RAG etc.

Despite their efficient design, IBM shared that Granite models don't sacrifice performance for cost. According to IBM’s benchmarks, Granite outperforms comparable models across various enterprise tasks, achieving high scores on Hugging Face's RAGBench Leaderboard. This targeted approach ensures organizations can select the right tools for specific business challenges without unnecessary computational overhead.

Multimodal Models: Expanding AI's Capabilities

Multimodal AI models can simultaneously process and interpret multiple types of data inputs—such as text, images, audio, and video. Unlike unimodal models that work with only one data type (typically text), multimodal models can understand the relationships between different forms of information, similar to how humans process the world through multiple senses.

IBM's Granite 3.2 vision models offer a practical application of multimodal capabilities in an enterprise context. Rather than focusing on image generation (creating pictures from text prompts), these models specialize in image understanding—extracting valuable information from visual content. At just 2 billion parameters, these specialized vision models can:

• Extract data from documents, even poorly scanned PDFs
• Analyze charts and graphs to answer specific business questions
• Process dashboard screenshots to provide insights on performance metrics
• Interpret receipts and other visual business documents

Making Strategic AI Decisions: The Performance-Cost Matrix

As AI technology evolves, organizations must balance cost, accuracy, and safety for model use. Soule shared at least four guidelines for evaluating what models to use within the enterprise, that include:

1. Right-sizing your models. Match model capabilities to business requirements rather than defaulting to the largest available.
2. Designing model selection criteria with transparency and governance in mind. Understand how models were trained and whether they align with your governance requirements.
3. Delivering systems-based security. Implement guardrails and safety protocols beyond relying solely on the model's built-in safeguards.
4. Understanding and supporting various customization needs. Assess how much adaptation a model will require for your specific use cases.

For example, Soule discussed how different models may offer various levels of transparency and governance. One distinguishing feature of IBM Granite models is that IBM publishes detailed information about their training datasets and methodologies, allowing enterprises to understand what's "under the hood" of these models. The Granite ecosystem includes risk and harm detection capabilities, transparency tools, and IP protection. Model transparency allows IBM to provide indemnification for Granite models, offering businesses additional protection when deploying these AI solutions.

The Shift is Underway

The "bigger is better" paradigm is giving way to a more nuanced approach to enterprise AI. Business leaders can achieve comparable performance by strategically implementing smaller, specialized models for appropriate use cases while significantly reducing costs and complexity. IBM's Granite models are one example of this approach. We’ve also seen the large foundation model providers, such as Open.AI and Meta, support smaller models.

Looking ahead, we're moving toward more flexible AI deployment models where businesses can dynamically allocate resources based on task importance. This could mean using models of various sizes for different tasks or enabling features like "reasoning" when demands such as accuracy justify the additional cost and latency. As AI becomes further integrated into business operations, this efficiency-focused approach will be increasingly critical for sustainable AI adoption and competitive advantage.

You can subscribe to our video channel here https://www.youtube.com/@AIwithMaribelLopez and our podcast here https://bit.ly/3R0Etal.