Solar Power and Energy Demand

This issue of Digital Davincis is based on the capstone project that I was a part of in my master in Data Science at UNSW

Executive Summary:

Project Overview

• Aim: Enhance Australia's energy demand forecasting.
• Team: Joshua Evans, Bernard Lo, Andrew Ryan.
• Result: Created a model outperforming existing solutions.

Collaboration and Workload

• Overcame time zone challenges with effective communication.
• Strategic task division based on strengths and experience.
• Tools: GitHub, Teams, OneNote, Python, Jupyter Notebook.

Project Takeaways

• Weather is the primary driver of electricity demand variability.
• V-shape relationship between temperature and electricity demand.

Forecasting Methods

• Mathematical: Time Series Analysis, Regression Analysis.
• AI: Machine Learning, Neural Networks.
• Econometrics: Cointegration Models, Structural Equation Modeling, Panel Data Analysis.

Importance of Solar Power

• Enhances forecasting accuracy as it reduces demand from traditional sources.
• Critical in regions with high solar adoption like Australia.

LSTM Model

• Ideal for time-series forecasting.
• Inspired by Google Translate’s use of LSTM.
• Captures long-term dependencies and contextual relationships.

Data Scraping

• Automated with Python’s requests library.
• Organized for accuracy and repeatability.

Results

• Models with solar data outperformed those without.LSTM with solar data had R2 value of 0.9739 (vs. 0.9662 without).
• Better predictions enable optimized supply, pricing, and reduced wastage.

Conclusion and Food For Thought

• Solar power data leads to more accurate forecasts.
• Future work: Explore additional features, more recent data, real-time forecasting frameworks, explore TimeGPT for transformer based predictions.

Introduction

Forecasting power demand is central for energy providers to optimize resources across different cities and accurately predict revenue from customers. In our Capstone project, my team (Joshua Evans, Bernard Lo, Andrew Ryan) and I aimed to enhance Australia's energy demand forecasting. In this accelerated 6 week project, we leveraged our diverse skills in data science and AI to bring a fresh perspective to the practice. Notably, our collaborative efforts resulted in a model that outperformed existing solutions on the market.?

Here is what I learned from this experience.

The Power of Collaboration

Our team, comprised of individuals from diverse skill sets, had a unique opportunity to create something remarkable. The main challenge we faced was synchronizing our efforts across various time zones, as each member was balancing professional responsibilities alongside their studies. Our project spanned continents, with members located in Buenos Aires, Sydney, Indonesia, and Singapore. This added a small challenge to overcome, which we solved by staying available on the group chats, working independently on technical part and reserve our stand up meetings to more high level concerns such as interpreting results and redefining the project’s direction.

Splitting the Workload

We allocated tasks based on each member's strengths and experience, ensuring a balanced team dynamic. I assumed the role of data engineer and programmer, focusing on technical aspects like coding and web scraping. Meanwhile, other members handled project management, data analysis, and report writing, leveraging their specific skills. This strategic division allowed us to effectively utilize our diverse expertise. Having talented and serious team mates gave me the confidence to work on my

Tools and Techniques for Efficiency

To streamline our workflow and communication, we utilized several tools: Github, Teams, OneNote and Python among others.

GitHub facilitated collaborative coding and version control, while Microsoft Teams enabled seamless communication across time zones with structured file organization. Microsoft OneNote was used for documenting research findings and meeting notes. Python and Jupyter Notebook were used for data analysis and exploration. The success of our group effort was mainly due to an effective exploratory data analysis, regular stand-up meetings, and a strategic task division. Future improvements would focus on enhancing our communication and adjusting task priorities to better align with project outcomes. Key ingredients for team success include clear communication, technology, task allocation, structured meetings, and mutual trust.

Project Takeaways

Electricity Demand and Weather

One might think that the main driver of power demand is price, but.Weather is the primary driver of electricity demand variability. Figure 1 below illustrates this by showing the average daily demand in New South Wales versus the average air temperature at Bankstown Airport from January 1, 2020, to December 31, 2020.

Source: UNSW ZZSC 9020 Data Science Project, Hexamester 2 , 2024

The scatterplot demonstrates higher electricity demand at both low and high temperatures, with lower demand at moderate temperatures, forming a striking V-shape. The bottom of the V sits around 18-19 degrees Celsius. Above 19 degrees, demand rises due to increased air-conditioning use, while below 18 degrees, demand increases as electric heaters are used.

Different Ways to Forecast Energy Demand

Forecasting energy demand is crucial for efficient energy management and planning. There are three main approaches to forecasting energy demand: Mathematical, Artificial Intelligence (AI), and Econometrics.

1. Mathematical Methods: Mathematical methods involve statistical models that use historical data to predict future energy demand. Common techniques include:
2. Artificial Intelligence (AI): AI-based methods leverage advanced computational techniques to predict energy demand with high accuracy. Some AI techniques include:
3. Econometrics Econometric: methods combine economic theory, mathematics, and statistical inference to model and predict energy demand. These methods focus on understanding the relationship between energy consumption and economic variables. Techniques include:

The Importance of Using Solar Power

As solar power becomes more prevalent, its impact on energy demand forecasting cannot be ignored. Integrating solar power production data into forecasting models can significantly enhance their accuracy. Solar power contributes to the overall energy supply, especially during peak sunlight hours, and can reduce the demand from traditional energy sources. By considering solar power production, energy providers can better predict actual demand, optimize energy distribution, and reduce wastage. This is particularly important in regions with high solar adoption, such as Australia, where rooftop solar panels are becoming increasingly common.

The Model We Chose: LSTM

Long Short-Term Memory (LSTM) models are a type of Recurrent Neural Network (RNN) designed to handle sequential data, making them ideal for time-series forecasting like energy demand. LSTMs can learn from data where important events are separated by long time lags, overcoming the vanishing gradient problem faced by standard RNNs. Interestingly, the same LSTM model architecture was used by Google Translate to handle the complexities of language translation, capturing long-term dependencies and contextual relationships between words. Inspired by its success in such a sophisticated application, we decided to use LSTM for our energy demand forecasting.

Scraping

Data collection was a critical part of our project. We used Python’s requests library to automate the retrieval of ZIP files containing energy and weather data from the web. By dynamically constructing URLs for each month and year within our study period, we efficiently downloaded the necessary datasets. Once downloaded, we unzipped these files using Python's zipfile library and organized them into structured directories for further analysis. This automation ensured accuracy and repeatability in data collection, eliminating human errors and ensuring comprehensive data coverage.

Results

Our analysis focused on the Queensland dataset, using both traditional and advanced modeling techniques to forecast electricity demand. The results were compelling:

• Models incorporating solar power data consistently outperformed those that didn't, across all error metrics (MSE, RMSE, MAE) and the coefficient of determination (R2).
• The LSTM model with solar data achieved an R2 value of 0.9739, compared to 0.9662 without solar data, demonstrating the significant improvement in accuracy.

These findings support the hypothesis that including solar power data enhances the accuracy of energy demand forecasts. The enhanced models provide better predictions, enabling energy providers to optimize supply, pricing, and reduce wastage. The success of this proof of concept in Queensland paves the way for applying the methodology to other states, potentially improving energy management across Australia.

Discussion

Our findings demonstrate the value of including solar power data in energy demand forecasting. While the improvement was most pronounced in Queensland, other states like Victoria and South Australia also showed better results with solar data, albeit to a lesser extent. This suggests that as solar adoption increases, its impact on energy forecasting will become even more significant.

Conclusion and Further Issues

Incorporating solar power production data into electricity demand forecasts leads to more accurate predictions. This improvement can help energy providers better manage supply, optimize pricing, and reduce wastage.

For future work, we recommend exploring additional features such as household energy efficiency, population growth projections, and the uptake of electric vehicles. Extending the analysis to more recent data and developing frameworks for real-time forecasting could further enhance predictive capabilities.