The Challenges and Opportunities of Liquid Cooling Solutions in AI Server (2)

Per my prior article (https://www.dhirubhai.net/pulse/challenges-opportunities-liquid-cooling-solutions-ai-server-rob-chang-a495c/?trackingId=sK1eSXeISl%2BT8A61t22nCg%3D%3D), thanks to my followers. It irritates a lot of interesting discussions.

Key Challenges for Liquid Cooling Adoption:

Liquid cooling offers distinct advantages over traditional air cooling, particularly in handling high thermal loads. Nevertheless, Qiu highlighted four critical hurdles that must be addressed:

1. Leakage Risks: Leakage remains a pressing concern, with instances of liquid cooling systems causing operational disruptions and customer dissatisfaction. Preventing leaks and defining liability in the event of failures are essential for broader acceptance.
2. Extending scope: Not only CPU and GPU, but other components may also require liquid cooling, such as AI accelerator (Smart NIC), HBM (Each HBM is 30W and 6 HBMs are 200W), and Power supply (Current design is around 5.5kW and the new design will be 85kW)
3. Increased Validation Complexity: Unlike well-established air cooling solutions, liquid cooling requires more rigorous validation processes. Suppliers like Auras and AVC have obtained NVIDIA certifications for liquid cooling components, underscoring the necessity for extensive testing and qualification.
4. Lack of Standardization and Modularization: The absence of standardized designs and modular components complicates procurement and integration for ODMs and cloud service providers (CSPs), slowing down adoption.
5. Extended Data Center Construction Timelines: Liquid cooling demands infrastructure upgrades, such as pipelines and chilled water units, which extend the construction timelines of data centers. On average, data center builds take 3–5 years, delaying the implementation of liquid cooling at scale.
6. Market Dynamics: From Air Cooling to Liquid Cooling: For decades, air cooling has dominated server thermal management. Even innovative designs, like IBM's water-cooled systems, failed to gain traction due to limited thermal requirements from earlier-generation chips. However, NVIDIA's GB200 server, set for limited production by late 2024 and mass production in Q1 2025, is changing the game.

The GB200 (1200W) incorporates liquid cooling as a standard feature, marking a paradigm shift. Current air-cooled systems like NVIDIA's Hopper AI servers, which use 3D vapor chamber (3D VC) technology, can only dissipate up to 750W. Liquid cooling, on the other hand, offers a heat dissipation capacity up to 28 times greater than traditional air cooling, making it indispensable for next-generation servers. No need to mention GB300, the requirement for heat dissipation is around 1400W.

Sustainability and Energy Efficiency

Beyond thermal performance, liquid cooling aligns with growing environmental sustainability goals. Data centers currently allocate up to 40% of their power consumption to cooling systems. Liquid cooling can reduce this proportion to just 7%, enabling more efficient energy utilization and significantly lowering carbon footprints.

Future Outlook

The demand for liquid cooling will surge with the advent of NVIDIA's GB300 servers, which will impose even higher cooling requirements. Despite these prospects, adoption rates remain low, with liquid cooling projected to account for only a single-digit percentage of the market by 2024. Scaling up by 2025 will require addressing the aforementioned challenges. CSPs are already accelerating investments in new data centers to support the transition. The integration of liquid cooling technology represents not just a technological shift but a redefinition of industry standards.