The Impact of Generative AI Workloads on Power, Cooling, Water, and Data Center Design: A Global View

For today’s blog I combined my last 10-years passion for AI with what I did early on (first 16 years) in my career in Energy & Power and see if I could bring it all together.? Artificial Intelligence (AI) is changing the world— but behind this magic lies a massive challenge: power, cooling, and water. AI doesn’t just need a computer; it needs entire cities’ worth of energy, giant cooling systems, and sometimes lakes of water to keep running. And not all AI tasks are the same—training a model is like building a skyscraper, while answering your question is like delivering a pizza. These differences shake up how we design data centers, where we put them, and how we power them.

In this blog, we’ll break down five key AI workloads—Training, Fine-Tuning, Retrieval-Augmented Generation (RAG), Prompt Engineering, and Inferencing—and explain how they gobble up power, heat up rooms, slurp water, and force us to rethink data centers. We’ll explore whether one data center can handle them all or if we need specialized ones. Then, we’ll map this to the U.S. power regions (like Texas or California) and zoom out to the world (Middle East, China, Europe) to see where these AI hubs should live based on energy, water, and climate. Buckle up—it’s a wild ride!

Understanding AI Workloads: How They Eat Power and Heat

Imagine an AI data center as a kitchen. Different recipes (AI tasks) need different tools, energy, and time. Here’s how each workload behaves:

Training - The Power-Hungry Beast

• What It’s Like: Teaching an AI everything from scratch—like raising a kid to be a genius. It takes months of non-stop work.
• Power Demand: Huge and steady. Think 90-100% of a GPU’s max power (like 700-1000 watts per GPU) for weeks or months. A 500 MW data center (powering 400,000 homes) might run flat-out for 84 days to train a big model like GPT-4.
• Power Pattern: A “step function”—it flips on full blast and stays there, like a factory running 24/7.
• Heat: Insane. Thousands of GPUs packed together get as hot as ovens—up to 150 kilowatts per rack!
• Water: Lots. Cooling this heat often uses water towers that evaporate millions of gallons yearly.

Example: Google’s Oklahoma data center trains models with wind power, but it still needs massive juice—think powering a small city.

Fine-Tuning - The Focused Tune-Up

• What It’s Like: Taking a smart AI and giving it a crash course in something specific—like teaching a chef to master sushi. Shorter but still intense.
• Power Demand: High (70-90% GPU use) but only for hours or days. Less than training but still heavy.
• Power Pattern: Bursts of high power, then rest—like a sprint instead of a marathon.
• Heat: Hot, but not as extreme as training. Still needs serious cooling.
• Water: Moderate. Less runtime means less water, but liquid cooling might still drink plenty.

Example: NVIDIA’s DGX systems fine-tune models for companies, using big power but not for long.

Retrieval-Augmented Generation (RAG) - The Librarian with a Brain

• What It’s Like: Combining AI with a giant search engine—like a librarian who reads every book to find your answer.
• Power Demand: Moderate (40-70% GPU use). Heavy when building the database, lighter when answering.
• Power Pattern: Spikes during indexing (like a busy day), then calmer—like a library during off-hours.
• Heat: Medium. Storage systems (like SSDs) add heat alongside GPUs.
• Water: Some. Indexing phases need cooling, but it’s not constant.

Example: Pinecone’s data centers build RAG systems for legal searches, balancing compute and storage.

Prompt Engineering - The Recipe Tweaker

• What It’s Like: Crafting perfect questions for AI—like tweaking a recipe until it’s just right. It’s lightweight work.
• Power Demand: Low (10-30% GPU use). It’s more about testing than heavy lifting.
• Power Pattern: Tiny bursts—like flipping a light switch on and off.
• Heat: Minimal. Barely warms the room.
• Water: Almost none. Air cooling is enough.

Example: OpenAI engineers tweak prompts on small systems, not big data centers.

Inferencing - The Quick Delivery Guy

• What It’s Like: AI answering your question in real-time—like a pizza guy racing to your door.
• Power Demand: Low to moderate (10-30% GPU use), but it spikes with traffic.
• Power Pattern: Spiky—like rush hour at a restaurant.
• Heat: Low but variable. More users, more heat.
• Water: Little. Air cooling often works, though busy times might need more.

Example: Amazon’s Alexa runs inference on edge data centers near you for fast replies.

What This Means for Data Centers: Power, Cooling, and Water Needs

Each workload is like a different customer at a buffet—they eat different amounts, make different messes, and need different setups. Here’s how they shape data centers:

Power Needs:

• Training: Needs a power plant’s worth of steady energy—like nuclear or coal (baseload power). A 500 MW center might need 345kV lines (big power highways) to avoid wasting 40 MW daily (enough for 32,000 homes).
• Fine-Tuning: Likes gas plants that ramp up fast (intermediate power) and high-voltage lines (138-345kV).
• RAG: Works with gas or solar plus batteries (intermediate), using 138-230kV lines.
• Prompt Engineering & Inferencing: Fine with quick-start gas or solar (peaking power) and medium-voltage lines (69-138kV).

Cooling Needs:

• Training: GPUs packed tight need liquid cooling—like pipes of cold water on chips—boosting power use by 25%. Think 3M Novec fluid or even liquid nitrogen for crazy heat (150kW/rack).
• Fine-Tuning: Similar but shorter, so hybrid air-liquid cooling works.
• RAG: Moderate heat needs air plus some liquid cooling for storage.
• Prompt & Inferencing: Simple air cooling—think big fans—since heat is low.

Water Needs:

• Training: Water towers evaporate millions of gallons yearly to cool heat—like a small lake disappearing.
• Fine-Tuning: Less time, less water—maybe thousands of gallons.
• RAG: Spikes during indexing use water, then taper off.
• Prompt & Inferencing: Barely a sip—air cooling means little water.

Layout:

• Training: Giant halls with dense GPU racks and pipes everywhere.
• Fine-Tuning & RAG: Mixed zones—compute plus storage areas.
• Prompt & Inferencing: Small, modular setups near users (edge centers).

All-Purpose vs. Specialized Data Centers: Can One Do It All?

Imagine a restaurant serving steak, sushi, pizza, and coffee. It’s possible, but the kitchen would need every tool, and the chef would be stretched thin. Same with AI data centers—here’s the debate:

All-Purpose Data Centers: Possible? Yes, with tricks:

• Power Zones: Separate areas for training (big power) and inferencing (spiky power).
• Smart Scheduling: Run training at night when power’s cheap, inferencing all day.
• Hybrid Cooling: Liquid for training, air for inferencing.
• Battery Backup: Smooths spikes for inferencing, supports training’s steady draw.

Examples:

• NVIDIA’s AI Factories: Handle training, inferencing, and more with massive setups.
• Microsoft’s Azure AI: Balances all workloads with dynamic power allocation.

Challenges: Expensive—think $500 million to build—and complex to manage.

Specialized Data Centers: Why? Each workload fits better with specific power, cooling, and locations:

• Training: Near power plants, huge substations, 345kV lines.
• Inferencing: Near cities, smaller transformers, 69-138kV lines.
• Fine-Tuning & RAG: Flexible spots with gas or solar, medium setups.

Examples:

• Google’s Training Hubs: Oklahoma, Wyoming—near wind or nuclear.
• Amazon’s Edge Centers: Urban areas for fast inference.

Trend: Specialization is growing—training hubs save power, inference edges cut latency.

What’s Best?

• All-purpose works for big players (Microsoft, NVIDIA) with cash and tech to juggle everything. But most are specializing—cheaper and simpler. Hybrid models (training hubs + inference edges) are popping up too.

Power Infrastructure: Transmission, Distribution, Substations, and Batteries

AI workloads don’t just need power—they need it delivered right. Think of it like water pipes:

Transmission (Power Highways):

• Training: 345kV+ lines—big pipes for huge flow. Saves $20M/year in losses per 500 MW center.
• Fine-Tuning: 138-345kV—smaller but still beefy pipes.
• RAG: 138-230kV—medium pipes for mixed loads.
• Prompt & Inferencing: 69-138kV—garden hoses for quick bursts.

Distribution (Local Pipes):

• Training: Giant transformers (75-150 MVA—like powering a city), thick feeder lines.
• Fine-Tuning & RAG: Medium-large transformers (33kV to 480V), reinforced feeders.
• Prompt & Inferencing: Standard transformers (10-25 MVA), regular feeders.

Substations (Power Hubs):

• Training: Dedicated, high-capacity transformers with backups (2N+2—like four spare tires). A 1-hour outage costs $208,000!
• Fine-Tuning & RAG: Industrial-grade with some redundancy.
• Prompt & Inferencing: Basic commercial setups—less risk if they fail.

Battery Storage:

• Training: Limited use—steady power trumps storage.
• Inferencing & RAG: Big help—Tesla Megapacks (4 MWh) smooth spikes, save solar for night.
• Fine-Tuning: Moderate—bridges short gaps.

Mapping AI Data Centers to U.S. Power Regions

The U.S. has power zones like ERCOT (Texas) or PJM (Virginia). Each has unique energy mixes—baseload (steady), intermediate (flexible), peaking (quick)—that fit different AI workloads:

ERCOT (Texas)

• Energy: 42% gas, 26% wind, 15% solar, 5% nuclear.
• Best For: Training (near gas/nuclear), Fine-Tuning (gas flexibility).
• Why: Isolated grid, fast permits, but wind/solar need batteries for steady loads.

Challenges: 345kV lines lag—West Texas renewables are 300 miles from Dallas AI hubs.

PJM (Virginia/Mid-Atlantic)

• Energy: Gas-heavy, some coal/nuclear, $5B grid upgrades.
• Best For: Training (Data Center Alley), Inferencing (near users).
• Why: 345kV lines coming, tax breaks, but gas reliance fights climate goals.

MISO (Midwest)

• Energy: Coal, gas, wind (4GW in North Dakota), $22B 765kV lines by 2032.
• Best For: Training (future baseload), RAG (wind + storage).
• Why: Cheap land, wind potential, but cold winters up heating costs.

CAISO (California)

• Energy: 57% renewables, 6GW batteries.
• Best For: Inferencing (edge near tech hubs), RAG (solar + storage).
• Why: Green power, talent, but earthquakes and rules slow builds.

NYISO/NEPOOL (Northeast)

• Energy: Hydro, nuclear, gas.
• Best For: Fine-Tuning (hydro/gas mix), Inferencing (urban edges).
• Why: Steady hydro, dense cities, but pricey land.

Going Global: Where AI Data Centers Thrive Worldwide

Now, let’s zoom out. Energy, water, and climate (heat, humidity) shape where AI data centers fit globally:

Middle East (e.g., Saudi Arabia, UAE)

• Energy: Oil/gas (baseload), growing solar.
• Water: Scarce—cooling is tough.
• Climate: Hot—needs advanced cooling (cryogenic?).
• Best For: Training (oil power, 345kV lines), but water limits scale.

Malaysia/Indonesia

• Energy: Coal, hydro, gas.
• Water: Plentiful—rivers and rain help cooling.
• Climate: Humid—air cooling struggles, liquid wins.
• Best For: Fine-Tuning, RAG (hydro/gas mix).

China

• Energy: Coal (60%), hydro, solar/wind growing.
• Water: Varies—north is dry, south has rivers.
• Climate: Mixed—cold north, humid south.
• Best For: Training (coal/hydro), Inferencing (urban edges).

India

• Energy: Coal, solar (fast growth), some hydro.
• Water: Spotty—monsoons help, droughts hurt.
• Climate: Hot—cooling costs soar.
• Best For: Inferencing (solar + batteries), RAG (mixed power).

Europe (e.g., Nordics, Germany)

• Energy: Nuclear, hydro, wind, solar.
• Water: Abundant—lakes and rivers aid cooling.
• Climate: Cool—less cooling needed.
• Best For: Training (hydro/nuclear), Fine-Tuning (green grids).

Putting It All Together: The Future of AI Data Centers

AI is rewriting the rules for power, cooling, water, and data centers. Here’s the big picture:

• Specialization Rules: Training hubs near power plants (Texas, Nordics), inference edges near cities (California, China), and hybrid Fine-Tuning/RAG spots in flexible zones (Midwest, Malaysia).
• All-Purpose Innovates: Big players like NVIDIA and Microsoft build Swiss Army knife data centers, but they’re pricey and rare.
• Power Evolves: 345kV lines, giant transformers, and mini nuclear reactors (like NuScale’s 77 MW units) are the future. Batteries help inference and RAG, not training.
• Cooling & Water: Liquid cooling (even nitrogen!) takes over for training, air works for inference. Water-rich spots (Europe, Southeast Asia) win.
• Locations Shift: Cheap power and cool climates (MISO, Nordics) grab training. Sunny deserts (Middle East, CAISO) with solar suit inference.

In the U.S., PJM and ERCOT lead now, but MISO’s 765kV lines by 2032 could steal the show. Globally, Europe’s green grids and Asia’s hydro-rich zones shine. One thing’s clear: AI’s hunger for power, cooling, and water isn’t slowing down—it’s forcing us to rethink everything. The winners? Places and companies that adapt fast to this $500 billion race.

One thing is for sure......thought my Energy and Power skills had rusted....with a bit of a brush up, it all started making sense!

Disclaimer: Above represent only my views and do not represent views of my current or past employers.