The Challenges and Opportunities of Liquid Cooling Solutions in AI Server (1)

As artificial intelligence (AI) continues to advance, its applications are increasingly computationally intensive, requiring vast processing power and generating immense heat. While people are working hard on reducing power consumption, traditional air-cooling methods, which use fans and airflow systems to dissipate heat, are reaching their limits in meeting the demands of high-density data centers where servers often run at full capacity. This challenge is particularly pronounced in AI servers, where workloads like deep learning, natural language processing, and complex model training require continuous, high-powered processing. The inadequacy of air cooling in handling such workloads (The theoretical limitation of air cooling in 1U servers is around 350-500W) has prompted data center engineers to consider alternative solutions that offer better heat dissipation and support the scalability needed for modern AI tasks. Following chart shows the TDP of AMD and Intel CPU is around 350-400W. No need to mention TDP of Nvidia GB200 is around 1200W.?

The adoption of AI across industries is driving exponential growth in computational demands, with AI workloads generating power densities up to five times higher than traditional data center workloads. This surge is pushing conventional cooling solutions to their limits, creating an urgent need for efficient cooling that can support these standards. While advanced air-cooling techniques, such as vapor chambers and 3D vapor chambers, offer temporary relief by enhancing heat dissipation at the chip level, they fall short of sustaining the high thermal demands of today’s AI servers. Vapor chambers, which utilize a sealed chamber filled with fluid to transfer heat via vaporization, and 3D vapor chambers (3D VC), which extend this concept to multi-layer designs for improved heat spreading, are useful for localized cooling but cannot scale to meet the full cooling requirements of high-density AI deployments because 3D VC requires at least 4U spaces.? Their limited cooling capacity and efficiency become evident as workloads increase, necessitating a shift toward liquid cooling for long-term sustainability. Liquid cooling, with its far superior thermal conductivity, offers a comprehensive solution by efficiently managing heat at both the component and rack levels, making it the preferred option for AI-driven data centers aiming for performance and scalability.

Challenges:?

One of the primary barriers to adopting liquid cooling in data centers is the significant upfront cost and infrastructure overhaul required. Unlike traditional air cooling, which relies on pre-existing HVAC systems, liquid cooling involves complex plumbing, specialized equipment, and, often, an entirely new infrastructure to support the circulation and management of coolant. Retrofitting an existing data center to accommodate liquid cooling can be costly, requiring extensive modifications to server racks, piping systems, and monitoring tools. Furthermore, the initial investment in liquid cooling systems, including high-quality pumps, heat exchangers, and cooling loops, can be prohibitive for many organizations. For companies with limited budgets, the cost challenge can lead them to favor short-term, air-based cooling enhancements like rear door heat exchangers (RDHX) only, even if those solutions cannot ultimately match liquid cooling’s effectiveness. As a result, companies must weigh the substantial initial costs of liquid cooling against the long-term performance and energy savings it promises.

Implementing liquid cooling also presents technical compatibility challenges, as it requires ensuring that server hardware (Such as the space of rack, chassis, and consideration of serviceability, reliability, and safety) and data center infrastructure (Such as chilled water availability, piping routes, spaces, etc.) are fully suited to liquid-based cooling systems. Not all server architectures are designed with liquid cooling in mind, and integrating these systems introduces risks, including potential leaks that could damage sensitive electronics. Data centers often need to upgrade or adapt servers, racks, and connectors to prevent coolant from interacting with electrical components, adding further complexity to the integration process. Additionally, liquid cooling systems must be tailored to work seamlessly with varying types of servers and different performance profiles within the AI environment, which can demand precise engineering and custom solutions. This compatibility challenge, along with potential leakage risks, underscores the need for robust safety protocols and compatibility testing, which add layers of complexity that many organizations are not yet prepared to manage.

Liquid cooling systems introduce new maintenance challenges that require specialized knowledge and resources, which can be daunting for data centers accustomed to the simpler upkeep of air-cooling systems. Unlike air-cooling setups, which primarily involve fan replacement and airflow management (Cubic feet per minute/CFM), liquid cooling requires routine monitoring of coolant quality, flow rates (Liter per minute/LPM), and temperatures. Additionally, the risk of leaks or coolant contamination necessitates regular inspection and preventive maintenance, often performed by trained technicians skilled in managing liquid-cooled systems. For many data centers, sourcing and retaining personnel with the expertise to handle these advanced cooling systems can be difficult, adding an operational burden that some may find hard to justify. Moreover, while liquid cooling systems are typically reliable, a malfunction could lead to significant downtime if not immediately addressed. This heightened need for specialized maintenance can be a deterrent for organizations without the necessary resources or personnel, reinforcing the complexity of adopting liquid cooling on a large scale.

Another set of hurdles for liquid cooling adoption involves environmental considerations and regulatory compliance. Some liquid cooling systems use specialized coolants that may contain chemicals requiring careful handling and disposal to avoid environmental harm. Managing these coolants responsibly adds complexity, as improper disposal could lead to environmental contamination and potential legal repercussions. Furthermore, regulations governing liquid cooling vary by region, with some areas imposing strict standards on coolant composition, waste disposal, and safety protocols. This regulatory landscape can make it challenging for global data centers to adopt a uniform liquid cooling strategy, as they must navigate varying rules and restrictions. For companies with sustainability goals, finding eco-friendly coolants and ensuring compliant disposal practices are essential yet complex tasks, adding to the operational burden of liquid cooling. Consequently, these environmental and regulatory challenges require companies to weigh the benefits of liquid cooling against the costs of meeting compliance and sustainability targets.

Opportunities:?

One of the most compelling benefits of liquid cooling in AI servers is its superior efficiency and the potential for significant performance gains. Unlike air cooling, which dissipates heat through fans and airflow, liquid cooling provides direct thermal contact with server components, enabling faster and more efficient heat removal. This efficient heat transfer allows AI servers to operate at higher power densities and process more intensive workloads without thermal throttling, which occurs when components overheat and automatically reduce performance to prevent damage. Liquid cooling can also reduce or even eliminate the need for traditional air conditioning, which is not only energy-intensive but often less effective for high-density deployments. By maintaining optimal temperatures, liquid cooling helps AI servers sustain peak performance, supporting advanced tasks like deep learning and real-time analytics. Nowadays except for the expensive immersion cooling technology, the major stream is a Direct-to-chip liquid cooling solution, which is a hybrid between liquid and air cooling. Even though machine efficiency gain is significant, the cost of liquid cooling solution is generally 15 times higher than air cooling solution. 4U 3D VC costs around $80 and it takes around 63% of overall air cooling modular. Coolant Distribution Unit (CDU) costs around $10k-30k and Cold plate costs around $200-300. These two critical components take around 85% of the overall Direct-to-chip liquid cooling cost.?

While the upfront costs of liquid cooling can be substantial, the long-term energy and cost savings make it a financially viable solution for many data centers. Traditional air cooling systems rely heavily on energy-intensive HVAC systems, which increase operational expenses as server densities rise. In contrast, liquid cooling’s efficient heat dissipation reduces the need for air conditioning, leading to substantial energy savings over time. Studies have shown that liquid cooling can decrease cooling energy costs by 30-40%, depending on the data center’s configuration and workload. These savings can offset the initial capital investment, providing a strong financial case for liquid cooling. Additionally, as energy costs continue to rise globally, the reduced power consumption of liquid cooling makes it a more sustainable and cost-effective solution for the future. For organizations looking to lower operational costs and improve their data center’s energy efficiency, liquid cooling presents an opportunity to achieve both objectives while accommodating the increased demands of AI applications. According to Nvidia’s analysis, based on the same computing power, the liquid cooling solution reduces 66% of rack space, reduces 28% of the power consumption, and PUE improves from 1.6 to 1.15.?

In addition to improved performance, liquid cooling can significantly extend the lifespan of data center equipment. Heat is one of the primary factors contributing to hardware degradation, as it accelerates the wear and tear of electronic components. By effectively managing temperature and reducing thermal stress, liquid cooling helps prevent overheating and mitigates the damage caused by temperature fluctuations, which are common with air cooling systems. Over time, this controlled environment minimizes the risk of hardware failures, reducing the frequency of costly repairs or replacements. In AI-focused data centers, where hardware investments are substantial, extending the useful life of servers and related equipment can lead to considerable cost savings. Furthermore, liquid cooling can improve operational uptime by reducing the chances of unexpected shutdowns due to overheating, ensuring AI workloads can run continuously without interruption. For companies aiming to maximize return on investment (ROI) in their data center infrastructure, liquid cooling offers a compelling advantage in equipment longevity.

Available option1:?

Direct-to-chip liquid cooling is one of the most widely adopted methods for managing heat in high-performance AI servers. This approach involves attaching cold plates directly onto the CPUs, GPUs, and other heat-generating components, allowing coolant to flow through the plates and absorb heat from these sources. Because the coolant directly contacts the hottest parts of the server, this method is highly efficient in dissipating large amounts of heat quickly. Direct-to-chip cooling is particularly advantageous for AI workloads, where processors often operate at high utilization for extended periods, generating continuous thermal loads that are difficult to manage with air cooling alone. While highly effective, direct-to-chip cooling requires careful system design and precision engineering, as components must be compatible with the coolant flow system to avoid leaks or performance issues. Despite these challenges, direct-to-chip cooling is an ideal choice for data centers looking to balance effective heat management with operational efficiency, especially in environments with moderate to high server density.

Available option2:?

Immersion cooling takes liquid cooling a step further by submerging entire servers in a specially engineered dielectric fluid that directly absorbs and transfers heat away from all components. This method is especially beneficial for data centers with ultra-high-density AI workloads, as it eliminates the need for traditional cooling infrastructure and allows for extremely compact server arrangements. In immersion cooling, the dielectric fluid, which is non-conductive and safe for electronics, circulates around the server, absorbing heat evenly and preventing hotspots. Immersion cooling is highly efficient and capable of handling power densities far beyond what air or direct-to-chip cooling can support. However, it also requires specialized infrastructure, from the tanks housing the servers to the circulation systems that manage the fluid flow and cooling. While immersion cooling offers impressive thermal performance, it’s a relatively new approach that can present challenges in terms of maintenance and fluid management, making it best suited for cutting-edge data centers willing to invest in experimental cooling technologies.

Available option3:?

Rear-door heat exchangers offer a hybrid approach that combines aspects of liquid and air cooling, providing an adaptable solution for data centers transitioning toward more advanced cooling methods. In this system, a heat exchanger is mounted at the back of each server rack, where it captures and dissipates heat from the servers as air is pulled through the rack. The liquid coolant flows through the rear-door heat exchanger, absorbing heat from the servers’ exhaust air before it can enter the room, thus reducing the need for extensive air conditioning. This hybrid cooling method is less invasive than direct-to-chip or immersion cooling and can often be integrated with existing infrastructure, making it a practical option for data centers not yet ready to commit to a fully liquid-cooled environment. Rear-door heat exchangers are also cost-effective for medium-density setups and can be combined with direct-to-chip cooling in high-performance areas, offering a flexible cooling solution that adapts to varying server densities and workload demands.?

Following is my experience of a notable example of successful liquid cooling adoption deployed by me. By faced with the challenge of scaling its data center infrastructure to meet rising AI demands, the company decided to implement direct-to-chip liquid cooling across several high-density server clusters. This transition required an overhaul of the company’s cooling infrastructure, which included retrofitting existing racks, installing custom cold plates, and training technicians in the maintenance of liquid cooling systems. Although the initial setup involved considerable investment, the company has since reported significant performance improvements and a 30% reduction in cooling energy costs. Additionally, the liquid cooling solution enabled the company to double its server density without exceeding its existing power budget. This case study demonstrates the transformative impact of liquid cooling on AI-driven data centers, particularly for organizations willing to make the initial investment in infrastructure and training to reap long-term efficiency gains.

One trend on the rise is the development of eco-friendly coolants that can be safely disposed of without causing environmental harm, addressing one of the primary regulatory and sustainability challenges associated with liquid cooling. Additionally, research into advanced materials, such as graphene-based or phase-change materials, is expected to increase the thermal conductivity of cooling solutions, making heat transfer even more effective. Modular liquid cooling systems, which can be added or removed depending on workload demand, are also emerging as a flexible option, allowing data centers to adjust cooling capacity dynamically. These trends indicate a shift toward more adaptable, environmentally responsible cooling solutions that can keep pace with AI advancements, enabling data centers to efficiently manage increased power densities without compromising on sustainability or cost.

Looking further ahead, novel cooling technologies such as liquid immersion combined with two-phase cooling are gaining attention as potential game-changers for ultra-high-density AI applications. Two-phase cooling, which involves a coolant that changes from liquid to vapor upon absorbing heat and then condenses back to liquid, offers even greater thermal efficiency by leveraging the energy-intensive phase change to absorb more heat than traditional cooling methods. This approach, when integrated into immersion cooling systems, has the potential to maximize cooling efficiency and drastically reduce power consumption, making it ideal for the most demanding AI data centers. Additionally, the integration of AI-driven thermal management systems, which monitor and optimize coolant flow, temperatures, and energy consumption in real time, could further enhance the precision and reliability of liquid cooling. These innovations reflect an industry moving toward smarter, more sustainable, and more scalable cooling solutions that can support the future needs of AI workloads.

Supply and sourcing strategy:

1. Supplier Selection and Evaluation:?

• Identify Specialist Suppliers: Begin by identifying suppliers with proven expertise in liquid cooling components, such as cold plates, pumps, heat exchangers, and dielectric fluids. Look for suppliers with a strong track record in high-performance computing (HPC) or data center cooling to ensure quality and reliability. If the buyer lacks resources and experience, selecting a liquid-cooling-solution integrator can be a good option, such as Foxconn. They are the most aggressive EMS focusing on integrated liquid cooling solutions including CDU, cold plates, manifolds, QC (Quick connector)/QD (Quick disconnector), and loops. Wiwynn directly invests Zutacore and aims to whole solution provider as well.
• Evaluate Supplier Capabilities and Innovation: Given the rapid evolution in liquid cooling, evaluate each supplier’s commitment to R&D and innovation. In the current supply market, there are three major supply groups in liquid cooling. EMS is the one, such as Foxconn, Wiwynn, Quanta, etc. The second group is Power supply vendors, such as Vertiv and Delta. The third group is existing thermal solution providers, such as Cooler Master, Nidec, and AVC. If the buyer is fresh to the market, starting with someone from these three groups can be a good selection. Btw, most importantly, their solution needs to be certified by Nvidia ^__^
• Assess Supply Chain Maturity: Ensure suppliers have robust supply chain practices, including redundancy for critical components and geographic diversity, to mitigate potential disruptions.
• Vendor Collaboration: Select suppliers open to collaborative product development. This is critical if customizations or integrations are needed to match the specific cooling requirements of AI servers or to stay agile for future innovations.

2. Cost Management Strategy

• Cost Modeling and Benchmarking: Use “Should Cost” modeling to assess the expected cost of components such as custom cold plates and immersion tanks. Benchmark costs against industry standards and competitors to ensure pricing remains competitive. Let me use a 2-piece cold plate as an example. Material: Retention bracket made of aluminum $2-4 per kg. Fluid heat exchanger made of copper $8-10 per kg. Manufacturing costs: Machining can account for 20-30% of the total cost of the cold plate due to labor, tooling, and machine wear. Brazing costs depend on the complexity of the channel structure but can represent 15-20% of the total production cost. Testing and assembly can account for 5-10% of the total cost, depending on the manufacturer’s quality control standards. The typical range of overhead and profit is around 10-20%.

• Total Cost of Ownership (TCO) Analysis: Consider the TCO by including not only upfront costs but also installation, maintenance, energy savings, and equipment lifespan extensions. This will help justify higher initial investments for long-term savings.
• Volume Pricing and Long-Term Contracts: Negotiate volume-based pricing and lock in costs through long-term contracts with suppliers of critical components. This approach can secure favorable pricing while mitigating the risks of price fluctuations, especially for specialty coolants.

3. Supply Chain Risk Management

• Diversification of Key Components: Diversify sourcing for critical components, such as pumps, heat exchangers, and specialty coolants, to avoid dependency on a single supplier. Consider alternative sources for each essential part of the liquid cooling system to ensure continuity.
• Geographic and Production Risk Mitigation: For global deployments, source from suppliers with facilities in different geographic regions to mitigate risks from geopolitical instability, climate events, or regional lockdowns.
• Supplier Relationship Management (SRM): Establish a strong SRM program to monitor supplier performance, inventory levels, lead times, and compliance with quality standards. Regular audits and performance reviews can identify potential risks early and allow for proactive issue resolution.

4. Scalability and Flexibility

• Modular System Design: Work with suppliers to design modular liquid cooling systems that allow for phased deployment as data center needs grow. This modular approach enables scalable solutions without a complete system overhaul, adapting to gradual increases in AI workload intensity.
• Support for Retrofit and Hybrid Systems: Select suppliers offering retrofit solutions for existing air-cooled setups and hybrid liquid-air cooling systems. This flexibility will allow for gradual transitions, helping manage costs and reduce operational disruptions.
• Future-Proofing with Emerging Technologies: Ensure suppliers have a roadmap for incorporating emerging cooling technologies like two-phase or immersion cooling. This will allow for a seamless upgrade path as cooling needs evolve and as more powerful AI processors require enhanced heat dissipation capabilities.

5. Environmental and Compliance Strategy

• Focus on Eco-Friendly Coolants: As environmental regulations tighten, prioritize suppliers who offer biodegradable or low-impact coolants to avoid costly compliance issues. Suppliers with closed-loop or recyclable systems may also align better with sustainability goals.
• Monitor Regulatory Compliance: Keep a close eye on regulatory changes around coolant composition, waste disposal, and emissions. Suppliers who proactively comply with or exceed these standards will reduce potential compliance risks.
• Circular Supply Chain Initiatives: Consider suppliers who support recycling or repurposing of components, such as coolant recovery and reusability programs, to reduce waste and support environmental sustainability efforts.

6. Long-Term Supplier Development and Innovation Partnerships

• Collaborate on New Cooling Technologies: Establish partnerships with suppliers and research institutions to co-develop next-generation cooling solutions tailored to AI servers. This collaborative approach can result in proprietary solutions that enhance cooling efficiency and competitive advantage.
• Supplier Training and Development: Invest in training suppliers on the specific requirements of AI workloads to ensure alignment on quality and performance standards. This strategy builds a stronger supply chain ecosystem, fostering innovation and reliability in liquid cooling solutions.
• Continuous Improvement Programs: Implement continuous improvement programs that enable suppliers to enhance system performance, energy efficiency, and component durability. Supplier performance incentives, such as tiered contracts, can motivate continuous innovation.