How Much Does It Cost to Self-Host an LLM? A Comprehensive Cost Analysis

?? TL;DR: Hosting the Llama-3 8B model on AWS EKS will cost around $17 per 1 million tokens under full utilization. Using GPT-4 Turbo costs $10 per 1 million prompt tokens and $30 per 1 million sampled tokens. If you choose to buy your own hardware, costs can drop to less than $0.01 per 1 million tokens, but it may take up to 5.5 years to break even at partial utilization.

1. Hosting on AWS EKS

I started by testing the Llama-3 8B model on AWS EKS. Initially, we tried a smaller Nvidia Tesla T4 instance (g4dn.2xlarge) but eventually switched to a more powerful g4dn.16xlarge with 4 Tesla T4 GPUs. This cut response times down to 5-7 seconds.

The cost of running this setup on AWS:

• $3.91 per hour or $2,816.64 per month for the g4dn.16xlarge with 4 GPUs.

After running calculations, I found that this setup can process around 157,075,200 tokens per month, which brings the cost to $17.93 per 1 million tokens under full utilization.

Pros of AWS EKS:

• Flexibility and scalability: You can dynamically scale resources up or down as needed.
• No upfront hardware costs: You only pay for what you use.
• Optimization: You have a chance to optimize costs by reserving instances. It's A good option if you load your model on a 24x7 basis.

Cons of AWS EKS:

• Relatively high cost: While AWS provides convenience and scalability, the cost of instances is still too high.
• Management: You have to manage the working time of your instances to save costs.
• Changes: Costs can change from time to time. IIt'sAWS)

2. GPT-4: The Convenience of a Cloud Solution

Now, let's look at GPT-4, particularly with its updated pricing:

• GPT-4 Turbo (128k context length) costs $10 per 1 million prompt tokens and $30 per 1 million sampled tokens.
• On average, this costs around $20 per 1 million tokens (prompt + sampled combined).

Example Cost Breakdown:

Let's say you send a query with 500,000 prompt tokens, and the model generates 500,000 sampled tokens. In this case:

• Five hundred thousand prompt tokens would cost $5.
• Five hundred thousand sampled tokens would cost $15.

Total: $20 for 1 million tokens.

Pros of GPT-4:

• No infrastructure management: Everything is set up and ready to use.
• Ease of use and flexibility: You can start with no upfront investment in hardware.
• Optimized infrastructure: You only pay for the tokens you actually process.

Cons of GPT-4:

• Pay-as-you-go model: The monthly cost can increase for large-scale or long-term usage.
• Less control over infrastructure: You can't fully customize the hardware or environment as you would with self-hosting.
• Security compliance: For some companies, keeping models out of the house is unacceptable.

3. Self-Hosting: Control and Long-Term Savings

For those who want full control over their infrastructure and are looking for long-term savings, self-hosting can be a good option. I explored a scenario where you buy 4 Nvidia Tesla T4 GPUs, which can be found for $700 each. The total upfront cost for the hardware would be:

• $2,800 for the GPUs.
• $1,000 for the rest of the setup (motherboard, power supply, cooling, etc.).

Total upfront cost: $3,800.

Monthly Operating Costs:

Electricity and maintenance costs are estimated at around $100 per month. If fully utilized, this setup could process 157,075,200 tokens per month, bringing the cost down to $0.00064 per 1 million tokens. That's a pretty good result and looks like the best option. But we have BUTs.

Pros of Self-Hosting:

• Low cost for high-volume usage: Once the hardware is paid for, the cost per token becomes incredibly low.
• Full control over the infrastructure: You can tailor the environment and model to your specific needs.

Cons of Self-Hosting:

• High upfront costs: The initial investment in hardware is significant.
• Long break-even period: If the hardware isn't used at full capacity, it could take up to 5.5 years to break even. Usually, companies load their models only on 20% of the working time.
• Maintenance responsibilities: You are responsible for managing and maintaining the hardware.
• Long recovery: If one of the parts of your server breaks, you will spend quite a long time recovering work on your project. As a result, you will pay more money and extend the project's payback period.
• Additional risks: Power outages, natural disasters, etc., can lead to unexpected expenses.

4. When to Choose What?

GPT-4: Quick Start with No Hassles

If you need to get up and running quickly without worrying about managing infrastructure, GPT-4 is a solid choice. It's ideal for businesses with unpredictable workloads, startups, or anyone looking to minimize upfront costs. This option is for those who experiment and need flexibility.

AWS EKS: Flexibility for Short-Term Use

AWS EKS offers flexibility and scalability without requiring you to buy your own hardware. It's useful for companies that need to scale up or down quickly. This option is relevant for those who care about compliance and require flexibility. There are ways to optimize costs in the cloud, but you must prepare yourself to invest more in the project.

Self-Hosting: Long-Term Savings with Full Control

If your project involves consistent, heavy usage, and you're willing to invest in the initial hardware setup, self-hosting can significantly reduce costs in the long run. However, it requires a significant upfront investment, risks, and ongoing maintenance.

Conclusion

Each solution has its own set of advantages and trade-offs, and the best choice depends on the specific needs and goals of your project.

• GPT-4 is great for simplicity and flexibility, with no upfront hardware investment.
• AWS EKS offers flexibility for scaling up resources without buying hardware.
• Self-hosting provides the lowest cost per token after the hardware is paid off, but requires a long-term commitment and significant upfront investment.

Finally, the choice between cloud-based solutions like GPT-4 or AWS EKS and self-hosting depends on your workload, budget, and operational needs. For many projects, starting with a GPT is the way to go, and then migrating to a cloud solution when your businesses need scaling. For others, considering self-hosting can be a balanced approach.

This article provides a neutral comparison of the available options, allowing you to make an informed decision based on your specific requirements. Whether you choose the ease of cloud-based solutions or the long-term savings of self-hosting, understanding the cost dynamics is key to optimizing your approach to LLM hosting.