Vast.ai

Today, we’re focusing on two powerful GPUs from NVIDIA: the L40 and L40S. While they may not have received as much attention as some of the company's other offerings, they certainly deserve a closer look! These two GPUs, built on the Ada Lovelace architecture, pack a punch in performance and capabilities.

LMDeploy is an open-source framework for Large Language Model inference. It is particularly good at high-throughput serving for multi-user or high-load use cases and is one of the most popular serving frameworks today. LMDeploy provides an OpenAI-compatible server, which means that it can be integrated into applications that use the OpenAI API. This makes it easy to switch from using OpenAI's services to running your own models on more affordable compute.

The world of Open Source AI has gotten many updates in the last month or so. There are now many new models with great quality:speed ratios and models that challenge the frontier of closed source models. This makes it even easier to build applications and automate workflows with open source models, which you can deploy on #VastAI, Meta, Mistral, and Nvidia have made the biggest waves with their recent releases. https://lnkd.in/g6bkw7gy

Setting up TGI to serve an LLM on #VastAI? We have you covered.

SGLang is a great beginning to building Generative AI Apps. Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins and shipping velocity. Using SGLang on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and state-of-the-art throughput of the SGLang backend.

It's a great time to be a developer using AI. In the past few months, there have been major capability improvements for LLM's and Multimodal models, enabling tasks that were not possible before with open source models. And the release of many smaller language models that are still as capable as previous generation models means that the cost curve is coming down for deploying to production. At Vast, we're very excited to see these advancements, and you can look out for more updates and building on top of these types of models for specific workflows in the coming months.

Embeddings are an important part of GenAI applications along with running inference of Generative Models. Now with vLLM, you can run them with the same docker image on Vast for ultimate flexibility.

As with any unconfirmed reports, it's best to take this news with a grain of salt – but the possibility has already sparked plenty of buzz among GPU enthusiasts.

In fact, here at Vast.ai, we're currently in the process of completing our SOC 2 Type 1 certification – further solidifying our commitment to data security and regulatory compliance. Our Compliance Policy is designed to protect your data every step of the way. Here's how:

The world of Open Source AI has gotten many updates in the last month or so. There are now many new models with great quality:speed ratios and models that challenge the frontier of closed source models. This makes it even easier to build applications and automate workflows with open source models, which you can deploy on Vast.ai. Meta, Mistral, and Nvidia have made the biggest waves with their recent releases. https://lnkd.in/g75iQUcs