Green AI: Sharing Language Models for a Sustainable Future

Transformers, despite their impressive capabilities, come at a significant environmental cost. Training these large models on massive datasets requires substantial computing power, leading to increased carbon emissions.

The below graph highlights the environmental impact of various human activities, emphasizing the need for more sustainable AI development.

Training Large Language Models (LLMs) is a computationally intensive process with significant energy costs.The type of energy used plays a crucial role in determining the environmental impact. While renewable energy sources can mitigate these concerns, reliance on non-renewable fuels like coal can exacerbate them.

Several factors contribute to the energy consumption of LLM training:

• Training Time: Longer training durations lead to increased energy consumption.
• Hardware Type: The computational power of the hardware used directly affects energy usage.
• Training Methodology: Practices like using pre-trained models, fine-tuning, and starting with smaller experiments can help optimize energy consumption.

To further reduce energy costs:

• Understand Hyperparameter Ranges: Knowing the typical ranges for hyperparameters can help minimize unnecessary experimentation.
• Employ Random Search: Random search often provides comparable results to grid search while using fewer resources.

By adopting these strategies, researchers and developers can contribute to more sustainable LLM training practices.

Imagine if each time your company wanted to train a model, it did so from scratch. This would lead to huge, unnecessary global costs!

This is why sharing language models is super important: Sharing the trained weights and building on top of already trained weights reduces the overall compute cost and carbon footprint of the community.

Following are some of the tools to find the carbon footprint of your activities

1. https://mlco2.github.io/impact/
2. https://codecarbon.io