Deep Reinforcement Learning (DRL) vs. Heuristics and Conventional Models: A Comparative Study

In the realm of energy optimization and control, selecting the right approach can have profound implications for efficiency, cost-effectiveness, and sustainability. One of the emerging contenders in the field is Deep Reinforcement Learning (DRL), a cutting-edge technique that holds the promise of revolutionizing the way we optimize distributed energy resources. In this article, we undertake a comparative study to evaluate DRL against heuristic methods and conventional optimization models.

The Challenge of Energy Optimization

Optimizing energy resources in a complex, ever-changing environment is a formidable task. Traditional optimization methods, such as Mixed Interger Linear Programming (MILP) or Dynamic Programming, often require a detailed representation of the energy system and have limitations when it comes to real-time, dynamic scenarios. This is where DRL comes into play. Its adaptability, ability to handle uncertainty, and capacity for learning from experience make it a compelling choice for energy management or automated trading.

To make a meaningful comparison, we embarked on a challenging case study. Our aim was to design a problem that would genuinely test the capabilities of DRL. We chose to model a simplified energy storage system tasked with capitalizing on price arbitrage in a volatile market using historical price data from EPEX Day-Ahead. The task was to determine when to charge (buy from the market) and discharge (sell to the market) the energy storage to maximize arbitrage.

Simplifications for Fair Comparison

To ensure an even comparison between DRL, heuristic methods, and conventional models, we made several simplifications in our case study:

1. Discrete Charge Levels: The energy storage was represented by only five discrete charge levels - 0%, 25%, 50%, 75%, and 100%.

2. Charging and Discharging Steps: Charging and discharging could only occur in increments of 25% of the full capacity.

3. Efficiency and Costs: We assumed an efficiency of 100%, and there were no associated costs with charging and discharging the storage.

While these assumptions may lead to unrealistically high potentials for arbitrage, they ensured that the problem could be implemented consistently across all three approaches. Moreover, the simplicity of the problem helped reveal performance differences more clearly.

Comparing Optimization Techniques

For our conventional model, we employed Dynamic Programming, which involved solving the Bellman Equation recursively. As a heuristic, we utilized a threshold model, defining two price thresholds: one for discharging at high prices and one for charging at low prices. This technique is equivalent to a often used minimal spread requirement between buy and sell. A gradient descent search was used to optimize these thresholds, reflecting the best results attainable with this method.

Theoretical considerations indicated that Dynamic Programming could find the optimal dispatch strategy, while the heuristic approach was expected to fall short of the optimal solution. DRL, on the other hand, had the potential to match the Dynamic Programming method on an in-sample evaluation. But what about the performance applying it out-of-sample to price data it had not seen so far.

Results

The results of our case study were illuminating:

- Conventional Model (Dynamic Programming): 100%

- Heuristic (minimal Spread): 49%

- DRL (out of sample): >95%

This data highlights the impressive capabilities of DRL. Notably, these results were obtained using an out-of-sample approach, meaning that DRL was evaluated on data from a different year than Dynamic Programming and the heuristic. This underscores the ability of DRL to generalize its learning across diverse datasets, a characteristic that conventional optimization methods lack.

Implications for Real-World Application

The results of this study have significant implications for real-world energy optimization. DRL's performance, even when faced with challenging out-of-sample data, demonstrates its potential to excel in dynamic and stochastic scenarios. This adaptability makes it a promising choice for real-time energy management where unforeseen events and deviations are common.

As compared to conventional techniques like MILP or DP we should also highlight the comparably fast computation speed of the DRL approach. The trained DRL model was able to calculate dispatch for the next 24 hours at a 50 Hz frequency (i.e. in intervals of 20 milliseconds) on a conventional business notebook, making it a good choice for complex real-time-optimization taks.

In conclusion, Deep Reinforcement Learning has shown great promise as a viable optimization technique for distributed energy resources. Its adaptability, capacity for learning, and ability to handle uncertainty position it as a valuable tool in the energy sector's ongoing transition towards more sustainable, decentralized, and efficient systems. While challenges remain, DRL's impressive performance in this comparative study is a testament to its potential to shape the future of energy optimization and control.