The costs of GenAI
Matt Carlin
Driving Innovation, Optimizing the Customer Experience, and Transforming Enterprises
The New York Times reported that “OpenAI expects to lose roughly $5 billion this year.”(). The coming years look substantially less dire, according to the article, as revenues tripled this year and are expected to top $11.6 billion next year. But, how does OpenAI expect to get there? One way could be from the announced a $200 a month 'Pro' tier - 10x the price of the existing 'Plus' subscription.
While increasing the revenue coming in the door is certainly needed, reducing expenses is another lever to consider in cutting back those operating losses. One factor that may be in OpenAI's favor going into 2025 is that they may be reaching Zero Marginal Cost to run their models. Today, let's look at the concept of zero marginal cost in relation to OpenAI, and more broadly across Generative AI.
What is Zero Marginal Cost?
In case you are unfamiliar, there is a financial concept that most of the cost of creating a product is incurred up front. If you are manufacturing medicine, you need the research to develop the drug as well as the equipment to actually make and package it. The old saying goes, “the first pill costs a billion dollars and the second costs fifty cents.” Zero marginal cost is the idea that after the big upfront investment, its nearly free to produce as much of your product as the market will stand. The upfront investment can then be averaged across everything you create, turning billions into pennies per product, or near zero cost.
Not every industry works this way. While manufacturing and digital products (think software or media) do exhibit near zero marginal cost, service-focused businesses do not enjoy that same scale. As a consultant, I can't clone myself and do 3x or 10x the work in the same amount of time. An hour of human effort is an hour of human effort. It can be enhanced with automation, but scaling up human effort involves bringing on more people, which increases costs and prevents a service-based business from reaching zero marginal cost.
What are the costs associated with GenAI?
As we turn to Generative AI, we can look at the different types of costs associated with developing and scaling this technology. There are substantial upfront costs, there are on-going costs that don't necessarily change with consumption, and then there are on-going costs that scale up and down based on product usage. Here is a non-exhaustive list:
One Time Fixed Costs
领英推荐
On-going Fixed Costs
Per-Use / Variable Costs
The various types of fixed costs can be divided down to near zero given sufficient scale, but what does that scale do to variable costs? This is the make or break category for zero marginal cost. Every time you interact with Gen AI, it consumes resources (electricity, sending data over the network, creating more data, etc). That is equivalent to the spend on raw materials that go into an aspirin or other pill. It typically wouldn't add a substantial amount to marginal cost.
There are two distinctions to make with GenAI though. Each use creates some data that can be fed back into the system, continuous fine-tuning is expensive but improves the model's output over time. The more the product is used, the more data gets fed into this fine-tuning process and more data makes the re-training more expensive. Additionally, new techniques like 'reasoning' generate more usage per interaction. Having the AI generate multiple responses and picking the one that shows up the most, having the AI double check its own work, or having the AI create a reasoning chain of thought and then work through that chain before giving an answer are all means of creating higher quality output. At the same time, the AI is essentially doing more work per interaction, consuming more resources and becoming more expensive, not less, to run at scale.
Does Zero Marginal Cost matter?
No industry ever really gets to 0. Software has traditionally been a zero marginal cost product because you write it once, then every copy after the first is essentially free. But, as we saw, service businesses usually have costs tied to service delivery and generally, more services delivered equals more costs incurred.
The costs of GenAI are a bit different than most software with increasing costs accruing at run time. It does start to feel more like a traditional service business. In that case, does Zero Marginal Cost matter? Businesses consume services, and given the return on investment projections, they may spend millions on a service. Whether or not GenAI can tamp down on its variable costs or restructure to make usage essentially free to run might not be the right question. Maybe the question should be, will the value created with this new class of AI outpace the spend? Organizations will need to make a critical judgment; where is the investment positively returning value and where is the proposition upside down? The market will sort this out in time, you either control your costs or spend yourself out of business.
Generative AI will continue to dominate headlines for the foreseeable future and its an exciting time to work with the technology given how many opportunities and possibilities it holds. Do you think AI will stay as cheap as it is today or will the continuous deployment of new models and techniques push the price up? Will we always compare AI to the cost of the same human labor or will other justifications be needed to determine the value of an AI project?