My crazy homelab-server build, an up to 1 TB RAM, up to 24 Petabyte storage and virtualization monster for AI testing.

TLDR: Built a homelab-server from refurbished and scrap parts for tinkering with virtualization and AI technology. Up to 1TB main memory, up to 24 Petabyte disk space – for under 1200 Euros. Just to feed the geek inside me. Not work related.

If you are looking for the business-related content, which I am usually posting here on linkedin – skip this article. This article is from a tech-geek for the tech-geeks out there, with a lot of tech jargon. POV and opinions are my own.

Prelude

During the past two decades, I ended up with three servers at home with a total capacity of about 40TB in various RAID configurations. The newest, but smallest one, has virtualization and Docker enabled but just 8GB of RAM and 2 disk-bays. The oldest one is a pure NAS and still runs SMBv1 as network protocol. In short, it is time for an upgrade.

Hence, a few months ago I decided to build a new homelab server during nights and on weekends – as sustainable and cost effective as possible, but with as much flexibility and scalability as I could possibly get. Because the new server must last at least for the next 10 years.

Tonight, I finished it.

System Choices

The obvious choice was a modern, yet two to three generations old workstation or server platform, where retired and refurbished parts are readily available on auction platforms or specialized IT-retailers.

One day I found my foundation layer – a scrapped x86 workstation, where half of the chassis was missing and cables hanging out, but the guts seemed ok. Enough PCIe-lanes, enough CPU cores and threads – at least for now. A perfect donor for my project.

Next was a new chassis – which I had to buy new unfortunately, because I needed a decent amount of 5.25 inch bays in the front and as many internal drive-bays as possible for what I was about to build. I found a 10+year old model for very cheap, because hardly any ‘modern’ home-computer tower-chassis still requires 5.25 inch bays.

Then we need more memory. The 32GB mounted in the donor, didn’t make the cut. I went for 8x32GB (256GB total) refurbished DDR4 RDIMMs, because these are more energy efficient than the up to 8x128GB LRDIMMs i.e. 1TB of main memory, which the donor supports.

Another problem was disks space. The design point was to reuse as many old drives as possible. I thought about integrating an NVME array, but that would have defeated the purpose of reusing old stuff. I had piles of disks from the old servers plus newer 2.5 inch HDD and SSD drives from various laptop migrations. I also shucked some old USB drives, yielding a few more Terabytes of spinning rust. All set – but how do we connect these?

The mainboard generously offers 4 slimSAS connectors, which I terminated into 16 SATA3 interfaces – so 16 disks can be directly connected to the mainboard. But you have probably guessed it by now, the geek in me just needs that little bit of extra headroom.

So I scavenged a popular auction platform. And one day there it was: a refurbished HBA (host bus adapter) capable of managing up to 1024 SAS or SATA drives. The geek inside me was delighted. If I’d connect current top-end 24TB drives, this would give me about 24 Petabyte of storage space. That’s the headroom for future expansion I was looking for. Ideal for software defined storage. Will I ever use it? Probably not. But I am using the additional “direct” 16 SAS/SATA connections by hooking them up via miniSAS HD (SFF-8843) with a 16-drive-bay cage for 2.5 inch drives mounted in two of the 5.25 inch bays.

Operating and File System Choices

Linux was a no brainer. The choice and setup of the disk arrays and file systems was much more difficult. I assessed XFS, BTRFS, OpenZFS as well as Ceph and GlusterFS. I took out Ceph and GlusterFS, because – for the time being – this will be a single-node configuration.

Between XFS, BTRFS and OpenZFS, I honestly haven’t made a final choice yet as all have their own advantages and disadvantages. So for the time being, I am using multiple arrays with different filesystems – and even within filesystems, I am using different setups like OpenZFS in single-drive, vdevs of 2 drives (mirror) and vdev of 4 drives (raidz1) configurations, depending on purpose.

A used 500GB NVME drive, which came with the donor, speeds up network and disk operations, because it serves as cache for the main array. -- And if you’d ask me about ZFS tuning like L2ARC, etc. Not yet. But it’s on the geek list ;-)

Workloads

The system is primarily built to run Docker and virtual machines (Linux & Windows) for various workloads including tinkering with AI – besides serving as backup storage server. - AI, you ask, don’t you need more compute power? Yes, my friend, but I’ll start with what I have right now and if that’s not enough, there are still four x16 PCIe slots available to add used graphic cards with ample tensor cores and do a PCIe passthrough to the virtual machines via the hypervisor.

Enough for now. - Let’s get the shell fired up and start tinkering and creating:

> docker create network Onwards

Inspiration, Epilogue and Next Steps

I was inspired for this project by many of my colleagues, who relentlessly innovate and develop ”geek” stuff, but in an industrial context. Bleeding edge applications and PoCs like Sven Strassburg with his AI-powered visual inspection of trains, or Russ McKay ’s and Martin Harris ’ “hammer rig” for preventing catastrophic asset failures by combining IoT and AI methodologies. And many others. - You are my north stars in Tech!

And for anyone, who has read this article so far – congratulations! And I need your help please: Every server needs a name. How would you call this one? Share with the tech enthusiasts in your network for even more ideas. :-)