Hypervisor Migration -> Small Form Factor!

Contents:

• What problem was I trying to solve?
• What have I done?
• Explanation of Physical Hardware Choices.
• How did I move my Hypervisor & Guests?
• What were the challenges?
• Closing thoughts...

What Problem Was I Trying to Solve?

Not long ago, I was sitting in my home office, studying, working, and the like, when I was struck with a fanciful and novel notion. "Why should I not strive for 'Peace and Quiet' in my home workspace, " I asked myself? Therefore, I set myself to the task of reducing the clatter and noise of my immediate environment. This created quite an enemy out of my rack-mounted Dell R510 Poweredge Server...

This server had long been my NAS / Proxmox Hypervisor mainstay. With nearly 200 days of concurrent uptime without a reboot or failure, it was certainly proving its longevity but, in view of my new goal, it was simply horrible. These old rack-mount units present the sonic equal of a vacuum in the next room over, humming and clicking incessantly 24/7 - 365. Furthermore, these units are HORRIBLY power-inefficient. According to my iDRAC Enterprise, my unit idles daily at over 250 Watts of draw from the wall. This is atrocious. Besides the fact that it often throws my room's circuit breaker, it produces heat, noise, and BILLS. At this point, I am sure that the problem has been defined and, at the risk of redundancy, I will present it clearly and concisely here once more:

"My Server was too loud, too expensive to run, and too warm for our AC to combat...I had to downsize, down-budget, and cool down!"

What Did I Do?

Put simply, I replaced the internals of a Dell Optiplex 3000 SFF with components to provide improved capacity, performance, and stability (on par with that of a low-end server). After the hardware swap was complete, I migrated my virtual Proxmox guests to the new host, preserving their configurations, data, and logs.

The new machine solves all of my problems, with minimal sacrifice. It maintains virtually identical, in many cases improved, performance to my Poweredge R510 while reducing noise, complexity, cost, and heat-output! This project has been a success story for the books.

Explanation of Hardware Choices:

CPU:

The Intel Core i5 - 12500, 6 Core 12 Thread was my choice of CPU. This chip offers me four major things...

1. Support for ECC memory. This is a major plus in my book for availability of server workloads. It helps my system to be more resilient to memory-related issues, bugs, and crashes. Though I am not currently utilizing the feature, having the upgradability was important to me.
2. Low TDP Under Load. This chip, being a SFF Desktop-grade CPU, is naturally tuned to not be power hungry. It is cool and efficient, while not compromising on feature set and power. In fact, on Passmark, this CPU maxes out at 30 watts less than my previous server CPU, the Xeon 5650, under load.
3. Improved Performance. This CPU absolutely CRUSHES my previous Xeon 5650 in terms of performance. The technology has simply improved so much since 2011. On Passmark, my new selection scores a respectable 19929 benchmark...compare this to the 5725 of my previous mainstay.
4. Integrated Graphics (Transcoding). This CPU comes with Intel's Integrated UHD 770 Graphical Processor. This iGPU is really nothing to write home about, but it DOES give me the ability transcode video in real time...a capability that will serve me nicely for future media-server projects.

Memory:

The Corsair VENGEANCE LPX DDR4 64GB (2x32GB) 3200MHz memory kit is what I decided on for memory upgrades. This memory kit gets me NEARLY to the 80GB memory capacity of my previous server. However, what it lacks in 16GB of capacity, it makes up for by being nearly 3 times as fast for data-transfer. This means that a smaller amount of memory is required, per guest, to actually achieve HIGHER levels of efficiency and performance (at least according to my subjective experience with the new offering). Furthermore, the included heat-sink spreaders on these DIMMs far improve upon the bare PCBs of the cheap memory sticks of my old server, allowing for better heat distribution.

Networking:

The 2 Port 2.5Gb PCIe Network Card, Dual LAN Port 2.5 Gigabit Ethernet Interface Adapter, with Realtek RTL8125B is what I decided on for networking. This inclusion brings the total number of interfaces on this machine up to 2 (this is an intentional choice). The native NIC is to be used for management, while the other two interfaces act as the WAN and LAN sides of my virtual firewall respectively. This allows me to isolate these types of traffic, and maintain a superior level of security. The 2.5Gb inclusion also insures that I am ready for a potential ISP upgrade, when that time does eventually come.

Storage:

My Storage choices came down to a decision of engineering. In the end, I settled with a three-tiered architecture to facilitate high availability virtual guests, and long-term storage:

• TIER 1: Crucial T500 1TB Gen4 NVMe M.2 SSD --> This storage device is to be used for the boot drive for Proxmox. Furthermore, its high speeds and superior efficiency and reliability make it suitable for extremely high-need VMs, such as my virtual Pfsense firewall.
• TIER 2: Western Digital 4TB WD Blue SA510 SATA Internal Solid State Drive SSD --> This storage device is high-capacity, while maintaining quality performance and efficiency. This is the main storage unit for virtual guests.
• TIER 3: This is simply an old server-grade HDD that I had lying around. Nothing special. Older, but reliable...clocking in at 2TB. This is what I call my COLD STORAGE. Backups are replicated HERE. This is where they stay until a Cron automation is in place to mirror them to my Raspberry-Pi's Samba Server.

This storage system seems to suite my needs well...despite having lost the hardware RAID functionality that I had on my Poweredge server, I think that this still affords me an acceptable level of redundancy and capacity. I am satisfied with such a trade :).

This hardware, though a downgrade in a few select ways, namely the lack of IPMI and hardware RAID, is ultimately a net UPGRADE. It has provided me with quiet, efficient, and low-cost server computing with NO noticeable loss in performance or capability. A win / win!

How Did I Move my Hypervisor & Guests?

This process was very straight forward. It boiled down to a few main steps. I will outline them below:

1. Confirm that my latest backups include all needed information and configurations.
2. Utilize the Secure Copy (SCP) command to move these backups from my old Proxmox host to my NEW proxmox host (already on network and installed). Drop them into the vzdump directory, where backups are stored.
3. Use Proxmox's live restore feature to spin up cloned VMs based off of those backup images.
4. Power DOWN the old guests as I power ON the new ones, sequentially.
5. Confirm functionality and power down old host.

Though seemingly complex, this task was deceptively simple! I had the guests backed up, moved over, and live within an hour's worth of work! The only difficulties that came along with this, were in the department of the Pfsense firewall, which had some interesting driver incompatibilities to troubleshoot. I shall detail these issues in the following section.

What Were the Challenges?

The challenges faced break down into two buckets: Software and Hardware. I will separate these out and discuss them below...

Hardware Issue: Not enough SATA - Power Connectors!

This issue cropped up early on! When opening the Dell Chassis, and inspecting the internals, I realized quickly that my plan for a 3-tiered storage solution would be challenged by the non-modular OEM power supply. This power supply included a single full-sized SATA - Power connectory...and a single SATA - Slimline power connector. Since LITERALLY NOTHING accepts the latter standard (apart from maybe a small DVD drive - No), I scoured the internet for a small business that makes adapters for PC internals. I finally found one here. This adapter allowed me to convert the smaller connector into a full sized SATA - Power connector, evading the compatibility issue entirely.

Hardware / Firmware Issue: BIOS Incompatibility with Hardware!

This issue instantly reared its ugly head the moment that I booted (or attempted to boot) my system for the first time. As it turns out, Dell's proprietary BIOSs are NOT out-of-the-box compatible with everything in the world...go figure! I got LUCKY on this one. After multiple reboots, some BIOS settings changes, and a live-boot session to pull down the latest BIOS image for flashing, I was able to gain full hardware compatibility. A crisis averted, that should have never been allowed to manifest. A quick check to Dell's compatibility metrics would have prepared me for this issue. Always check first!

Software Issue: Pfsense Firewalls are INCOMPATIBLE with Realtek NICs!

This issue was a fun one. As it turns out, Realtek and Pfsense are somewhat of enemies. Upon booting up my shiny new Firewall VM, I saw no interfaces! This was odd, since I was sure that I passed them through. After browsing the forums I was met with the sad truth. Pfsense does not natively support Realtek NICs. Luckily, there is always a solution...depending on how clever you want to be!

The first option would be to pull down a community-driven driver package for Realtek NICs from the FreeBSD repositories. These would prove to work but only marginally. They offered patchy performance, at best, and, because of their community driven nature, were not reliable. I was scared of the idea that a rogue software update or repository recompilation could brick my system and prevent me from reaching the internet. As such, I passed on this option after testing.

The other, BETTER, option was to utilize paravirtualization, a recently supported Pfsense feature, to pass through the NICs. This offers me the ability to utilize my NICs are nearly their full hardware bandwidth capacity, while sidestepping the compatibility issue. Although paravirtualization IS a true form of virtualization, it only translates the instruction sets for functions that require translation. Otherwise, it allows the guest to access the device directly. As a result, I get nearly full operating performance, while still presenting the device as a "vNIC" to Pfsense...an agnostic component that it supports by default, as the drivers are handled at the Hypervisor level.

Closing Thoughts:

This project was NOT my longest, most technically complex, or most resume-worthy project. But it was FUN. I thoroughly enjoyed diving back into the world of hardware-specifications and hardware troubleshooting. I needed, and enjoyed, the refresher that this project gave me in the server administration realm as well. It is always nice to know that I am able, when called upon, to rough my way through a low-downtime migration of virtual guests from one host to another. It keeps me sharp. It also trains me into good habits, like keeping recent backups, having redundant storage, and putting real engineering thought into my configurations and deployments.

Proxmox is an amazing product, and I had a blast standing it up on yet ANOTHER server of mine (this makes FOUR that I have done). I cannot wait to keep using this build for projects into the future! I have some ideas :) and I cannot wait to share them!

Thank you for reading, if in fact you did read this.

Tyler Sell