Building a PC that can run AIs

Perfecting Equilibrium Volume Three, Issue 23

And nothin' gonna stop me

Ain't no stop and go

I'm speedin' on the midway

I gotta really burn this road

Speedin' on the freeway, gotta get a lead way

Speed demon

Doin' it on the highway, gotta have it my way

Speed demon

Mind is like a compass, I'm stoppin' at nothin'

Speed demon

Yes, pull over, boy, and get your ticket right

The Sunday Reader, Nov. 17, 2024

I am given to odd habits and hobbies, but perhaps the most disconnected is building a PC every half-decade or so. I’ve built all my desktops for the last few decades, plus PCs for family members who wanted them.

I guess that’s a lot of machines, but it’s disconnected because computers last a while, so it’s three to five years between builds. And those long gaps between builds means the technology has had time to change quite a bit, so even rebuilding a PC I’ve had forever seems like starting from scratch for the very first time.

I have a couple of general rules of thumb for builds:

• Last year’s second-best technology is the value sweet spot
• For long-term components buy the best you can afford
• Take whatever the current recommendation is for RAM and double it

These rules have served me well, allowing me to sail along for years without ever seeing any little hourglasses or other “working on it” signs from my computer. But eventually they start to show up, and I start researching upgrades.

Buy Me a Cup of Coffee

I got almost five years out of Red Demon Mark II, then things went south in a hurry. For years my most demanding software has been Adobe Photoshop, and Red Demon Mark II never sweated photo editing.

Then Adobe started loading Photoshop with AI tools, and the PC started struggling. Around that same time I started doing a lot of video editing.

And then I started running Large Language Models – “AIs” – on Red Demon, and one thing was immediately clear:

Time to upgrade!

This is the third iteration of Red Demon, named for its red Cooler Master case and speed demon performance. Here are the builds:

I’ve added cores and threads, and quadrupled the RAM. How fast is Red Demon Mark III? Fast enough that the PhotoShop AI tools such as Remove Background are now instantaneous. Fast enough that exporting an eight-minute video from CapCut has gone from taking enough time to go and make a sandwich down to just a few seconds.

I followed my usual rules with the upgrades:

• The 7900X has most of the performance of the 7950X while being cheaper and requiring less power. And the release of the Zen 5 Ryzen series made it last year’s second-best CPU, and great sales were available. I was able to get an excellent deal on the 7900X with an excellent motherboard and a 32-gigabyte RAM kit from Microcenter. I added a second 32-gigabyte RAM kit and I was good to go.
• I always spend about the same amount on drives. The Seagate Barracuda 7200 RPM 1 terabyte hard drive was state of the art when I bought it for around $200, as was the 256 gigabyte SSD I replaced it with a half-decade later. I was planning to buy a fast 2-terabyte NVME for my new primary drive, but then I noticed that the motherboard that came in the bundle had on-board support for RAID 1. That means instead of having a single, fallible main drive, it could use a second as a mirror backup.
• The older drives stay as storage, which is one of the advantages of a giant tower case

I ended up spending double what I normally do on upgrading the graphics card. I enjoy a bit of light gaming; mostly stuff like Mechwarrior. Who doesn’t love big stompy robots whacking each other? And PhotoShop certainly benefits from the hardware. I’ve found that buying whatever graphics card is in the $150-$200 range has been perfect for the last few builds. That range generally hits the sweet spot of the second or third-best tech from the last generation.

No such luck this time, for a couple of reasons. First, graphic card prices have gone completely batpoop crazy. I’d been buying those $200 cards when the high-end ones were going for $400 to $600. Now that gets you into the lower end of the midrange. And only if you shop hard. The real high-end cards are up in the four figures. NVIDIA’s top-of-the-line 4090 cards are $2,000. And its rumored replacement, the 5090, is supposed to debut in January for $2,500.

And no, these are pro cards.

Those cost a lot more.

Pro cards used to be focused on workstation tasks such as CAD/CAM. While CPUs are generalists that can handle any sort of computer algorithm, GPUs are specialized for the type of math used to generate vector graphics and ray tracing for video games. But it turns out that specialization makes them superstars for several extremely difficult computer tasks, including mining for cryptocurrency tokens and training Large Language Models for use as AIs.

Mining drove demand and the price of GPUs sky high. Training AIs has driven their price clear into outer space.

For example, Elon Musk has built a supercluster in Memphis to train AIs for SpaceX, Tesla and X that uses NVIDIA H100 GPUs, which cost $25,000 each.

The Memphis cluster has 100,000 H100s.

This explains why NVIDIA’s market cap is $3.48 trillion this weekend; it is currently the world’s most valuable company, just ahead of Apple and Microsoft, the other two members of the $3 trillion market cap club.

And it explains why I had to lay out so much for a new graphics card. I’m working on a project running a Large Language Model on Red Demon; when it comes to that, NVIDIA is currently the only serious player.

There are GPUs from AMD Radeon and Intel that can seriously game for far less money, but if you want to efficiently run and train LLMs locally, NVIDIA is your only real choice. NVIDIA started supporting LLMs before anyone had heard of them, and now most such projects are optimized for NVIDIAs Cuda architecture.

When I started upgrading Red Demon this spring the “cheapest” NVIDIA GPU that could really handle local LLMs was the 4080, which was still an eye-watering $1,000. So I held off until the fall, when I was able to pick up an NVIDIA RTX 4070 Super, which offers three quarters of the LLM performance of the 4080 for half the price.

Buying the 4070 Super meant it was finally time to replace the original 550-watt power supply, which couldn’t produce the minimum 700 watts recommended for the new GPU. But it was time, anyway: a decade is a long time for a power supply.

So I’m set for the next few years, God willin’ and the creek don’t rise. And I can get back to work on building, training and testing a local LLM as a virtual editorial staff to support local reporting, at which the new rig really shines.

It's just a coincidence that it’s also amazing at generating movie-quality graphics for giant stompy robot games. Who knew?