5 Open LLM Inference Platforms for Your Next AI Application

Open large language models, like GPT-4 and Gemini, are a good substitute for commercial LLMs because of their growing capabilities. Developers are thinking about using APIs to access the most advanced language models because AI accelerator hardware can be costly. Although the obvious choices include cloud platforms like Google Cloud Vertex AI, Amazon Bedrock, and Azure OpenAI, there are also purpose-built systems that are less expensive and faster than the hyperscalers.

These five generative AI inference platforms, which include Mistral, Gemma, and Llama 3, can be used to consume open LLMs. Additionally, a few of them back foundation models that focus on vision.

1. Groq

The fastest AI inference technology in the world is allegedly being built by the AI infrastructure startup Groq. The Language Processing Units (LPU) Inference Engine, a hardware and software platform designed to provide AI applications with remarkable compute speed, quality, and energy efficiency, is their flagship product. Because of Groq's performance and speed, developers adore it. The GroqCloud service, which allows users to use popular open-source LLMs like Meta AI's Llama 3 70B at (allegedly) up to 18x quicker rates than other providers, is powered by a scalable network of LPUs. To utilize the API, you can use the OpenAI client SDK or the Python client SDK for Groq. Groq can be easily integrated with LlamaIndex and LangChain to create sophisticated LLM applications and chatbots.

In terms of pricing, Groq offers a range of options. For their cloud service, they charge based on tokens processed — with prices ranging from $0.06 to $0.27 per million tokens, depending on the model used. The free tier is a great way to get started with Groq.

2. Perplexity Labs

Perplexity is quickly taking the place of Google and Bing as an alternative. Although its main offering is an AI-powered search engine, Perplexity Labs also offers an inference engine.

Perplexity Labs unveiled pplx-api, an API intended to speed up and streamline access to open source learning materials, in October 2023. With pplx-api, users with a Perplexity Pro subscription can use the API, which is now in public beta. This allows a large user base to test and offer comments, assisting Perplexity Labs in continuously improving the tool. Popular LLMs such as Mistral 7B, Llama 13B, Code Llama 34B, and Llama 70B are supported by the API. As stated by Perplexity Labs, it is intended to be economically viable for both deployment and inference, with notable cost reductions. The OpenAI client-compatible interface allows users to easily integrate the API with pre-existing apps, which is beneficial for developers who are already acquainted with OpenAI's ecosystem. Check out my Perplexity API tutorial for a brief summary.

Based on the FreshLLM paper, the platform also contains llama-3-sonar-small-32k-online and llama-3-sonar-large-32k-online. These Llama3-based models have the ability to return citations; this functionality is presently in closed beta. For its API, Perplexity Labs provides a customizable price structure. consumers can utilize the pay-as-you-go plan without making any upfront commitments, as it charges consumers based on the quantity of tokens handled. The $20/month or $200/year Pro subscription offers unlimited file uploads, specialized assistance, and a $5 monthly credit toward API access. Depending on the scale of the model, the price per million tokens varies from $0.20 to $1.00. Online models are subject to an additional flat fee of $5 per thousand requests on top of the token expenses.

3. Fireworks AI

With the help of the generative AI platform Fireworks AI, developers may use cutting-edge open source models in their applications. It provides a large selection of language models, such as the vision-language model FireLLaVA-13B, the instruction-following models Mixtral MoE 8x7B and 8x22B, the Llama 3 70B model from Meta, and the function-calling model FireFunction V1. Stable Diffusion 3 and Stable Diffusion XL are two image-generation models that Fireworks AI supports in addition to language models. Fireworks AI's serverless API, which offers industry-leading throughput and performance, can be used to access these models.

The pricing structure of the platform is competitive. It provides a pay-as-you-go pricing model that takes into account the quantity of tokens processed. As an illustration, the Mixtral 8x7B model costs $0.50 per million tokens, but the Gemma 7B model costs $0.20. Additionally, Fireworks AI offers on-demand deployments, allowing customers to hire GPU instances (A100 or H100) by the hour. Because the API is OpenAI compliant, integrating it with LangChain and LlamaIndex is simple. With varying price points, Fireworks AI targets developers, companies, and enterprises. While the Business and Enterprise tiers include specialized rate limitations, team collaboration capabilities, and dedicated support, the Developer tier provides a rate restriction of 600 requests/min and up to 100 deployed models.

4. Cloudflare

With just a few lines of code, developers can execute machine learning models across Cloudflare's global network using the inference platform called Cloudflare AI Workers. It offers a scalable and serverless solution for GPU-accelerated AI inference, freeing developers from the burden of managing infrastructure or GPUs to use pretrained models for a variety of applications, such as speech recognition, image identification, and text synthesis. An extensive range of popular open-source models covering various AI activities is provided by Cloudflare AI Workers. Notable models supported are mistral-8x7b-32k-instruct, llama-3-8b-instruct, gemma-7b-instruct, and even vision models like segformer-b5-finetuned-ade-512-pt and vit-base-patch16-224.

For the purpose of developing new apps or adding AI capabilities to already-existing ones, Cloudflare AI Workers provides flexible integration points. To execute AI models within their applications, developers can make use of Workers, Pages Functions, and Cloudflare's serverless execution environment. A REST API is offered, allowing inference queries from any programming language or framework, for individuals who would rather integrate with their present stack. With Cloudflare's Vectorize (a vector database) and AI Gateway (a control plane for managing AI models and services), developers may improve their AI applications. The API supports activities including text generation, image categorization, and audio recognition.

With an affordable solution for AI inference, Cloudflare AI Workers offers a pay-as-you-go pricing approach depending on the number of neurons processed. Neurons function as a token-like unit because the platform offers a wide range of models that extend beyond LLMs. Every account has a free tier that permits 10,000 neurons daily, each of which aggregates usage from many models. After that, Cloudflare bills $0.011 for every 1,000 more neurons. The price is different for each model size. For example, Llama 3 70B costs $0.59 for every million input tokens and $0.79 for every million output tokens, whereas Gemma 7B costs $0.07 for every million tokens for input and output combined.

5. Nvidia NIM

Access to numerous pretrained big language models and other AI models that are accelerated and optimized by Nvidia's software stack is made possible through the Nvidia NIM API. Developers can examine and test out more than forty models from Nvidia, Microsoft, Hugging Face, and other sources using the Nvidia API Catalog. These include vision models like Stable Diffusion and Kosmos 2, as well as potent text-generation models like Microsoft's Mixtral 8x22B, Nvidia's Nemotron 3 8B, and Meta's Llama 3 70B.

With only a few lines of code, developers can quickly and simply incorporate these cutting-edge AI models into their applications thanks to the NIM API. The models allow for easy integration because they are housed on Nvidia's infrastructure and made available via a defined API that is compatible with OpenAI. Using the hosted API, developers may test and prototype their applications for free. When the models are ready for production, they can choose to deploy them on-site or in the cloud using the recently released Nvidia NIM containers.

Nvidia provides both free and paid tiers for the NIM API. The free tier includes 1,000 credits to get started, while paid pricing is based on the number of tokens processed and model size, ranging from $0.07 per million tokens for smaller models like Gemma 7B, up to $0.79 per million output tokens for large models like Llama 3 70B.