What is the hardware (cost) to fine tune an AI Model? A comparison of various models to date.

We all know that training an AI model from scratch is expensive.?

While that is prohibitive, how much does it cost to fine tune an AI model? While there are different cloud services with different price points, all rely on hardware (GPU and VRAM mostly) you need to choose to run or fine tune your chosen mode.

I will give some examples in this post of hardware for well known models.

Meta Llama 2

In August 2023 Meta released Llama 2, at that time the largest open source model to date. The largest version has 70b parameters. As an example, it is possible to fine tune Llama 2 in a single A100 with 80GB of VRAM with a dataset of 50k prompts, each of which is ~1000 - 1500 tokens of prompt + 500 tokens of response. It can be performed roughly in 4 days of training or 200$.

To fine tune the 7b version instead, on a 1k dataset with a single RTX series 3, it takes 3-4h.

For the 30b or 65b versions, you would need between 150 and 300 hrs respectively using about 72GB VRAM and 3 GPUS (series 3).

StableDiffusion

Using a Stability AI StableDiffusion 1.5, about 200 images, it takes about 20 min with a RTX series 3 and 24GB VRAM.

Google Gemma?

Google recently released 2b and 7b Gemma LLMs models (Feb 24). No precise tests I can report but it is safe to say that a quantized (i.e. reduced) version of the 7b model will require up to 8GB and A100 gpu. While the 2b model can run on a T4.

Grok1

Finally the latest and largest model released to date (March 24): Grok-1 ,314b parameters.

No clear indications yet but without quantisation we are talking about 1-2TB or more and therefore multiple top end GPUs (10-20).

That said, since it took less than a month to quantise Gemma models, I would say within 1-2 months we would have quantised Grok-1 fine tunable into GPU with 24-60GB VRAM.

Let’s see.

#ai #artificialintelligence #business #innovation #technology