As computers continue to get more powerful with CPUs, GPUs, NPUs, storage and memory they will require more power and run hotter. Cooling, especially liquid cooling becomes even more important and has a number of advantages:
- More cooling efficiency with lower TCO
- Less Noise than air cooling
- Higher density of compute in the same physical space
How does Cray supercomputer cooling work?
Cray supercomputers have employed various innovative cooling technologies over the years to manage the significant heat output generated by their high-performance computing components. Here’s an overview of the key cooling methods used in Cray supercomputers:
1. Liquid Cooling
Direct Liquid Cooling (DLC):
- How It Works: Liquid cooling involves the use of a liquid coolant, often water or a special coolant fluid, that is brought into direct contact with the heat-generating components (like processors and memory modules). The coolant absorbs heat and carries it away from the components.
- Components: Cold Plates: These are attached to the components. Coolant flows through these plates, absorbing heat. Pumps and Heat Exchangers: The heated coolant is circulated to heat exchangers where it transfers its heat to a secondary cooling system (often another liquid loop or an external chiller). Chillers: External units that further cool the liquid before it is recirculated back to the supercomputer.
- Efficiency: DLC is highly efficient as it directly removes heat from the source, reducing the need for large-scale air conditioning systems.
- How It Works: Components are submerged in a dielectric liquid that directly absorbs heat.
- Advantages: Excellent heat transfer efficiency and the ability to handle higher thermal loads compared to air cooling.
2. Air Cooling
Although liquid cooling is more common in modern Cray supercomputers, air cooling is also used, especially in earlier models and in conjunction with liquid cooling for less critical components.
- How It Works: Large fans or blowers move air across heat sinks attached to the heat-generating components.
- Components: Heat Sinks: Metal structures attached to components to increase the surface area for heat dissipation. Ducting and Airflow Management: Carefully designed paths to direct cool air to and hot air away from components.
3. Hybrid Cooling
Some Cray systems use a combination of liquid and air cooling to optimize performance and efficiency. For example, the most heat-intensive components might use direct liquid cooling, while other parts of the system use air cooling.
4. Evaporative Cooling
Evaporative cooling is another advanced technique that might be used in supercomputers:
- How It Works: It uses the principle of water evaporation to remove heat. Warm water from the cooling system is evaporated in a cooling tower or similar device, and the heat is carried away with the vapor.
- Advantages: Highly efficient and environmentally friendly as it can use less energy compared to traditional air conditioning.
Case Study: Cray XC Series
The Cray XC series supercomputers, such as the Cray XC40, utilize advanced liquid cooling systems:
- Liquid Cooled Door System: This system uses a rear-door heat exchanger that allows the system to be cooled using facility water.
- Direct Liquid Cooling Modules: These modules are directly attached to processors and other critical components.
- Scalable Cooling Infrastructure: The system is designed to scale with the cooling demands of the supercomputer, ensuring efficient operation at various loads.
Summary
Cray supercomputers employ a variety of cooling techniques, with a strong emphasis on liquid cooling due to its superior efficiency in managing high thermal loads. The combination of direct liquid cooling, hybrid systems, and advanced airflow management ensures that Cray supercomputers can operate at peak performance without overheating, maintaining reliability and extending the lifespan of their components.
How does the Elbrus supercomputer do cooling?
The Elbrus supercomputer series, developed by the Russian company Lebedev MCST, employs sophisticated cooling techniques to manage the heat generated by its high-performance components. Here’s an overview of the cooling methods used in Elbrus supercomputers:
1. Liquid Cooling
Direct Liquid Cooling (DLC):
- How It Works: Similar to many high-performance computing systems, Elbrus supercomputers use direct liquid cooling to efficiently remove heat from the system. Coolant, typically water or a specialized coolant fluid, is circulated through cold plates attached to the heat-generating components such as CPUs, GPUs, and memory modules.
- Components: Cold Plates: These are in direct contact with the components. The coolant absorbs heat from the components as it flows through these plates. Pumps and Heat Exchangers: The coolant is then pumped to heat exchangers, where it transfers the absorbed heat to a secondary cooling loop or directly to the facility's cooling system. Chillers: These units are used to further cool the liquid before it is recirculated back into the system.
2. Air Cooling
- How It Works: Forced air cooling is often used in conjunction with liquid cooling to manage less critical components or to provide additional cooling support. Air is moved across heat sinks attached to the components using large fans or blowers.
- Components: Heat Sinks: Metal structures that increase the surface area for heat dissipation, attached to components. Fans/Blowers: These create airflow to move cool air across the heat sinks and expel hot air from the system. Ducting and Airflow Management: Carefully designed airflow paths ensure efficient cooling by directing cool air to the hot components and expelling warm air out of the system.
3. Hybrid Cooling
Combination of Liquid and Air Cooling:
- How It Works: Elbrus supercomputers often employ a hybrid cooling system that combines both liquid and air cooling to optimize performance and efficiency. Critical components such as CPUs and GPUs may be cooled using direct liquid cooling, while other components are cooled using forced air cooling.
- Advantages: This approach provides the benefits of liquid cooling for high thermal load components while utilizing air cooling for less demanding components, resulting in an overall efficient cooling system.
4. Evaporative Cooling
Although not specifically documented for the Elbrus series, some high-performance computing systems use evaporative cooling techniques:
- How It Works: Evaporative cooling involves the use of water evaporation to remove heat. Warm water from the cooling system is evaporated in a cooling tower, and the heat is carried away with the vapor.
- Advantages: This method is highly efficient and environmentally friendly, as it can use less energy compared to traditional air conditioning systems.
Case Study: Elbrus-16S Supercomputer
The Elbrus-16S, one of the more recent models in the Elbrus series, utilizes advanced cooling technologies to ensure efficient thermal management:
- Liquid Cooling System: The Elbrus-16S employs direct liquid cooling for its processors and other critical components. This ensures that the heat is efficiently removed from the most heat-intensive parts of the system.
- Integrated Cooling Infrastructure: The system is designed with an integrated cooling infrastructure that can scale with the computational load, ensuring efficient cooling at various performance levels.
Summary
Elbrus supercomputers use a combination of direct liquid cooling and forced air cooling to manage the heat generated by their high-performance components. Direct liquid cooling provides efficient thermal management for the most heat-intensive components, while air cooling is used for additional support and less critical components. This hybrid approach ensures that Elbrus supercomputers can operate at peak performance while maintaining reliability and extending the lifespan of their components.
Sidenote: The Elbrus was designed to use cooling with distilled water, but they will tell you the backup liquid was Vodka.
How does supermicro handle cooling on its HPC/AI supercomputers?
Supermicro handles cooling for its HPC (High-Performance Computing) and AI (Artificial Intelligence) supercomputers using a variety of advanced cooling technologies designed to efficiently manage the substantial heat output from these powerful systems. Supermicro promotes liquid cooling that does not require datacenter retrofit or redesign, existing datacenters can be utilized with Supermicro liquid cooling servers. Here's an overview of the cooling methods employed by Supermicro:
1. Direct Liquid Cooling (DLC)
- Cold Plates: Supermicro uses direct liquid cooling by integrating cold plates that come into direct contact with the processors and other high-heat components. Coolant circulates through these plates, absorbing heat.
- Coolant Circulation: The heated coolant is pumped away to a heat exchanger, where it releases its heat before being recirculated back to the cold plates.
- Pumps and Heat Exchangers: High-efficiency pumps and heat exchangers ensure effective heat transfer and dissipation.
- Liquid Coolant: Often water or a specialized coolant.
- Cold Plates: Attached to CPUs, GPUs, and memory modules.
- Pumps: Maintain the flow of coolant.
- Heat Exchangers: Transfer heat from the coolant to the facility's cooling system or an external chiller.
2. Air Cooling
- Fans and Blowers: Supermicro uses high-performance fans and blowers to move air across heat sinks attached to heat-generating components.
- Heat Sinks: Metal structures that increase surface area for heat dissipation.
- Heat Sinks: Mounted on CPUs, GPUs, and other components.
- High-Efficiency Fans: Create airflow to dissipate heat.
- Airflow Management: Careful design of ducting and airflow paths to maximize cooling efficiency.
3. Hybrid Cooling Systems
Combination of Liquid and Air Cooling:
- How It Works: Hybrid systems combine the benefits of both liquid and air cooling. Critical components (like CPUs and GPUs) use direct liquid cooling, while less critical components are cooled with air.
- Advantages: Provides efficient cooling for high-thermal-load components while also being cost-effective and versatile.
4. Immersion Cooling
- Submersion in Dielectric Liquid: Components are submerged in a dielectric (non-conductive) liquid that directly absorbs heat from the components.
- Heat Dissipation: The heated liquid is circulated to a heat exchanger or cooling tower.
- High Efficiency: Excellent heat transfer capabilities.
- Noise Reduction: Immersion cooling is quieter than traditional air cooling.
5. Advanced Airflow Management
- Optimized Chassis Design: Supermicro designs its server chassis to optimize airflow, ensuring that cool air reaches the hottest components and hot air is efficiently expelled.
- Temperature Sensors: Integrated temperature sensors dynamically adjust fan speeds and cooling mechanisms to maintain optimal operating temperatures.
6. Liquid Cooling Kits and Custom Solutions
- Pre-Configured Kits: Supermicro offers pre-configured liquid cooling kits for certain server models, providing customers with a turnkey solution for liquid cooling.
- Tailored Cooling Solutions: For large-scale deployments, Supermicro works with customers to design custom cooling solutions that meet specific requirements, ensuring maximum efficiency and performance.
7. Case Study: Supermicro’s SuperBlade Systems
- Liquid Cooling Options: Supermicro’s SuperBlade systems offer optional direct liquid cooling for high-density server configurations. A liquid cooled GPU server can fit in a 4U chassis while the same GPU server air cooled would be in a 10U rack
- Integrated Cooling Infrastructure: These systems integrate both air and liquid cooling to provide scalable and efficient cooling for various workloads, including HPC and AI applications.
Summary
Supermicro employs a range of cooling technologies to manage the heat generated by its HPC and AI supercomputers, including direct liquid cooling, air cooling, hybrid cooling systems, and immersion cooling. Each of these methods is designed to ensure efficient heat dissipation, enabling Supermicro systems to operate at peak performance while maintaining reliability and extending component lifespan. Their approach includes pre-configured liquid cooling kits, optimized chassis design for airflow management, and custom cooling solutions tailored to specific customer needs.
How does Dell handle cooling on its HPC/AI supercomputers?
Dell handles cooling for its HPC (High-Performance Computing) and AI (Artificial Intelligence) supercomputers using a variety of advanced and efficient cooling technologies. Here's an overview of the cooling methods employed by Dell:
1. Direct Liquid Cooling (DLC)
- Cold Plates: Dell uses direct liquid cooling by integrating cold plates that come into direct contact with processors and other high-heat components. Coolant circulates through these plates, absorbing heat from the components.
- Coolant Circulation: The heated coolant is pumped away to a heat exchanger, where it releases its heat before being recirculated back to the cold plates.
- Pumps and Heat Exchangers: High-efficiency pumps and heat exchangers ensure effective heat transfer and dissipation.
- Liquid Coolant: Often water or a specialized coolant.
- Cold Plates: Attached to CPUs, GPUs, and memory modules.
- Pumps: Maintain the flow of coolant.
- Heat Exchangers: Transfer heat from the coolant to the facility's cooling system or an external chiller.
2. Air Cooling
- Fans and Blowers: Dell uses high-performance fans and blowers to move air across heat sinks attached to heat-generating components.
- Heat Sinks: Metal structures that increase surface area for heat dissipation.
- Heat Sinks: Mounted on CPUs, GPUs, and other components.
- High-Efficiency Fans: Create airflow to dissipate heat.
- Airflow Management: Careful design of ducting and airflow paths to maximize cooling efficiency.
3. Hybrid Cooling Systems
Combination of Liquid and Air Cooling:
- How It Works: Hybrid systems combine the benefits of both liquid and air cooling. Critical components (like CPUs and GPUs) use direct liquid cooling, while less critical components are cooled with air.
- Advantages: Provides efficient cooling for high-thermal-load components while also being cost-effective and versatile.
4. Immersion Cooling
- Submersion in Dielectric Liquid: Components are submerged in a dielectric (non-conductive) liquid that directly absorbs heat from the components.
- Heat Dissipation: The heated liquid is circulated to a heat exchanger or cooling tower.
- High Efficiency: Excellent heat transfer capabilities.
- Noise Reduction: Immersion cooling is quieter than traditional air cooling.
5. Advanced Airflow Management
- Optimized Chassis Design: Dell designs its server chassis to optimize airflow, ensuring that cool air reaches the hottest components and hot air is efficiently expelled.
- Temperature Sensors: Integrated temperature sensors dynamically adjust fan speeds and cooling mechanisms to maintain optimal operating temperatures.
6. Custom Cooling Solutions
Tailored Solutions for Large Deployments:
- Custom Design: Dell works with customers to design custom cooling solutions tailored to specific requirements, ensuring maximum efficiency and performance.
- Scalable Solutions: Dell's cooling solutions are designed to scale with the size and needs of the deployment, from small clusters to large supercomputing environments.
Case Study: Dell EMC PowerEdge Servers
Dell EMC PowerEdge Servers:
- Direct Liquid Cooling: Some Dell EMC PowerEdge servers come with optional direct liquid cooling solutions for CPUs and GPUs, enhancing cooling efficiency.
- Enhanced Air Cooling: These servers also feature advanced air cooling technologies, including high-performance fans and optimized airflow designs.
- Adaptive Cooling: PowerEdge servers use real-time thermal management and adaptive cooling algorithms to dynamically adjust cooling based on workload demands.
Summary
Dell employs a range of cooling technologies to manage the heat generated by its HPC and AI supercomputers, including direct liquid cooling, air cooling, hybrid cooling systems, and immersion cooling. These methods ensure efficient heat dissipation, enabling Dell systems to operate at peak performance while maintaining reliability and extending component lifespan. Dell also offers custom cooling solutions tailored to specific customer needs, providing scalable and efficient cooling for various deployment sizes and workloads.
How does HPE handle cooling on its HPC/AI supercomputers?
Hewlett Packard Enterprise (HPE) employs a variety of advanced cooling technologies to manage the significant heat output from its HPC (High-Performance Computing) and AI (Artificial Intelligence) supercomputers. Here's an overview of the cooling methods used by HPE:
1. Direct Liquid Cooling (DLC)
- Cold Plates: HPE uses direct liquid cooling by integrating cold plates that come into direct contact with high-heat components such as CPUs and GPUs. Coolant circulates through these plates, absorbing heat.
- Coolant Circulation: The heated coolant is pumped away to a heat exchanger, where it releases its heat before being recirculated back to the cold plates.
- Pumps and Heat Exchangers: High-efficiency pumps and heat exchangers ensure effective heat transfer and dissipation.
- Liquid Coolant: Often water or a specialized coolant.
- Cold Plates: Attached to CPUs, GPUs, and memory modules.
- Pumps: Maintain the flow of coolant.
- Heat Exchangers: Transfer heat from the coolant to the facility's cooling system or an external chiller.
2. Air Cooling
- Fans and Blowers: HPE uses high-performance fans and blowers to move air across heat sinks attached to heat-generating components.
- Heat Sinks: Metal structures that increase surface area for heat dissipation.
- Heat Sinks: Mounted on CPUs, GPUs, and other components.
- High-Efficiency Fans: Create airflow to dissipate heat.
- Airflow Management: Careful design of ducting and airflow paths to maximize cooling efficiency.
3. Hybrid Cooling Systems
Combination of Liquid and Air Cooling:
- How It Works: Hybrid systems combine the benefits of both liquid and air cooling. Critical components (like CPUs and GPUs) use direct liquid cooling, while less critical components are cooled with air.
- Advantages: Provides efficient cooling for high-thermal-load components while also being cost-effective and versatile.
4. Immersion Cooling
- Submersion in Dielectric Liquid: Components are submerged in a dielectric (non-conductive) liquid that directly absorbs heat from the components.
- Heat Dissipation: The heated liquid is circulated to a heat exchanger or cooling tower.
- High Efficiency: Excellent heat transfer capabilities.
- Noise Reduction: Immersion cooling is quieter than traditional air cooling.
5. Advanced Airflow Management
- Optimized Chassis Design: HPE designs its server chassis to optimize airflow, ensuring that cool air reaches the hottest components and hot air is efficiently expelled.
- Temperature Sensors: Integrated temperature sensors dynamically adjust fan speeds and cooling mechanisms to maintain optimal operating temperatures.
6. Custom Cooling Solutions
Tailored Solutions for Large Deployments:
- Custom Design: HPE works with customers to design custom cooling solutions tailored to specific requirements, ensuring maximum efficiency and performance.
- Scalable Solutions: HPE's cooling solutions are designed to scale with the size and needs of the deployment, from small clusters to large supercomputing environments.
Case Study: HPE Apollo Systems
- Direct Liquid Cooling: HPE Apollo systems, such as the HPE Apollo 2000 and Apollo 6000, offer direct liquid cooling options to improve cooling efficiency and performance density.
- Enhanced Air Cooling: These systems also feature advanced air cooling technologies, including high-performance fans and optimized airflow designs.
- Adaptive Cooling: Apollo systems use real-time thermal management and adaptive cooling algorithms to dynamically adjust cooling based on workload demands.
Case Study: HPE Cray EX Supercomputers
- Direct Liquid Cooling: The HPE Cray EX supercomputers use an advanced liquid cooling system where processors and other critical components are cooled directly by liquid, allowing for higher density and performance.
- Immersion Cooling: For some specific use cases, immersion cooling can be used to achieve exceptional thermal management.
Summary
HPE employs a range of cooling technologies to manage the heat generated by its HPC and AI supercomputers, including direct liquid cooling, air cooling, hybrid cooling systems, and immersion cooling. These methods ensure efficient heat dissipation, enabling HPE systems to operate at peak performance while maintaining reliability and extending component lifespan. HPE also offers custom cooling solutions tailored to specific customer needs, providing scalable and efficient cooling for various deployment sizes and workloads
How does IBM handle cooling on its HPC/AI supercomputers
IBM employs a variety of advanced cooling technologies to manage the significant heat output from its HPC (High-Performance Computing) and AI (Artificial Intelligence) supercomputers. Here’s an overview of the cooling methods used by IBM:
1. Direct Liquid Cooling (DLC)
- Cold Plates: IBM uses direct liquid cooling by integrating cold plates that come into direct contact with high-heat components such as CPUs and GPUs. Coolant circulates through these plates, absorbing heat.
- Coolant Circulation: The heated coolant is pumped away to a heat exchanger, where it releases its heat before being recirculated back to the cold plates.
- Pumps and Heat Exchangers: High-efficiency pumps and heat exchangers ensure effective heat transfer and dissipation.
- Liquid Coolant: Often water or a specialized coolant.
- Cold Plates: Attached to CPUs, GPUs, and memory modules.
- Pumps: Maintain the flow of coolant.
- Heat Exchangers: Transfer heat from the coolant to the facility's cooling system or an external chiller.
2. Air Cooling
- Fans and Blowers: IBM uses high-performance fans and blowers to move air across heat sinks attached to heat-generating components.
- Heat Sinks: Metal structures that increase surface area for heat dissipation.
- Heat Sinks: Mounted on CPUs, GPUs, and other components.
- High-Efficiency Fans: Create airflow to dissipate heat.
- Airflow Management: Careful design of ducting and airflow paths to maximize cooling efficiency.
3. Hybrid Cooling Systems
Combination of Liquid and Air Cooling:
- How It Works: Hybrid systems combine the benefits of both liquid and air cooling. Critical components (like CPUs and GPUs) use direct liquid cooling, while less critical components are cooled with air.
- Advantages: Provides efficient cooling for high-thermal-load components while also being cost-effective and versatile.
4. Immersion Cooling
- Submersion in Dielectric Liquid: Components are submerged in a dielectric (non-conductive) liquid that directly absorbs heat from the components.
- Heat Dissipation: The heated liquid is circulated to a heat exchanger or cooling tower.
- High Efficiency: Excellent heat transfer capabilities.
- Noise Reduction: Immersion cooling is quieter than traditional air cooling.
5. Advanced Airflow Management
- Optimized Chassis Design: IBM designs its server chassis to optimize airflow, ensuring that cool air reaches the hottest components and hot air is efficiently expelled.
- Temperature Sensors: Integrated temperature sensors dynamically adjust fan speeds and cooling mechanisms to maintain optimal operating temperatures.
6. Custom Cooling Solutions
Tailored Solutions for Large Deployments:
- Custom Design: IBM works with customers to design custom cooling solutions tailored to specific requirements, ensuring maximum efficiency and performance.
- Scalable Solutions: IBM's cooling solutions are designed to scale with the size and needs of the deployment, from small clusters to large supercomputing environments.
Case Study: IBM POWER Systems
- Direct Liquid Cooling: IBM POWER systems, such as the IBM POWER9, offer direct liquid cooling options to improve cooling efficiency and performance density.
- Enhanced Air Cooling: These systems also feature advanced air cooling technologies, including high-performance fans and optimized airflow designs.
- Adaptive Cooling: POWER systems use real-time thermal management and adaptive cooling algorithms to dynamically adjust cooling based on workload demands.
Case Study: IBM Blue Gene/Q
- Water Cooling: The IBM Blue Gene/Q supercomputer uses an advanced water cooling system where processors and other critical components are cooled directly by water, allowing for higher density and performance.
- Energy Efficiency: This cooling method significantly reduces energy consumption compared to traditional air cooling.
Summary
IBM employs a range of cooling technologies to manage the heat generated by its HPC and AI supercomputers, including direct liquid cooling, air cooling, hybrid cooling systems, and immersion cooling. These methods ensure efficient heat dissipation, enabling IBM systems to operate at peak performance while maintaining reliability and extending component lifespan. IBM also offers custom cooling solutions tailored to specific customer needs, providing scalable and efficient cooling for various deployment sizes and workloads.
What types of cooling employed on IBM Watson?
IBM Watson, particularly in its more advanced and resource-intensive configurations, utilizes several cooling technologies to manage the significant heat output from its powerful computing components. The cooling methods employed by IBM Watson include:
1. Direct Liquid Cooling (DLC)
- Cold Plates: Direct liquid cooling involves attaching cold plates directly to heat-generating components such as CPUs and GPUs. The coolant circulates through these plates, absorbing heat.
- Coolant Circulation: Heated coolant is pumped away to a heat exchanger where it releases the absorbed heat before being recirculated back to the cold plates.
- Pumps and Heat Exchangers: High-efficiency pumps and heat exchangers facilitate effective heat transfer and dissipation.
- Liquid Coolant: Often water or a specialized coolant.
- Cold Plates: Attached to critical components like CPUs and GPUs.
- Pumps: Maintain the flow of the coolant.
- Heat Exchangers: Transfer heat from the coolant to the facility's cooling system or external chillers.
2. Air Cooling
- Fans and Blowers: High-performance fans and blowers move air across heat sinks attached to heat-generating components, aiding in heat dissipation.
- Heat Sinks: Metal structures with a large surface area that facilitate the transfer of heat from the components to the air.
- Heat Sinks: Mounted on processors, memory modules, and other components.
- High-Efficiency Fans: Create airflow to dissipate heat effectively.
- Airflow Management: The chassis and internal design are optimized to ensure efficient airflow, directing cool air to hot components and expelling hot air out of the system.
3. Hybrid Cooling Systems
Combination of Liquid and Air Cooling:
- How It Works: Hybrid cooling systems leverage both liquid and air cooling. Critical components like CPUs and GPUs might use direct liquid cooling, while other components are managed with air cooling.
- Advantages: This approach balances efficiency and cost, ensuring high-performance components are adequately cooled without unnecessary complexity for less critical components.
4. Advanced Airflow Management
- Optimized Chassis Design: The design of the server chassis is optimized to ensure efficient airflow, making sure that cool air reaches the hottest components and hot air is expelled efficiently.
- Temperature Sensors: Integrated temperature sensors help dynamically adjust fan speeds and cooling mechanisms to maintain optimal operating temperatures.
5. Immersion Cooling
While not typically associated with IBM Watson, immersion cooling is a cutting-edge technology that IBM has explored in other contexts and could be applied to high-density configurations.
- Submersion in Dielectric Liquid: Components are submerged in a non-conductive dielectric liquid that directly absorbs heat from the components.
- Heat Dissipation: The heated liquid is circulated to a heat exchanger or cooling tower for heat removal.
- High Efficiency: Excellent heat transfer capabilities.
- Noise Reduction: Quieter operation compared to traditional air cooling.
Summary
IBM Watson employs a mix of cooling technologies to handle the thermal demands of its high-performance computing tasks. These include direct liquid cooling for critical components, air cooling with optimized airflow management for additional and less critical components, and potentially hybrid cooling solutions that combine both methods. These strategies ensure that IBM Watson can operate efficiently and reliably, maintaining optimal performance even under heavy computational loads.
