From Data to Decisions: Navigating the Energy Transition

In a recent interview with the ABC's Daniel Mercer about the curtailment of utility solar and wind in the National Electricity Market (NEM) [1,2], he posed a question that has stuck with me: “Why is data so important now?” This question, discussed in the lead-up to our interview, led me to reflect on my own journey in the energy sector.

An early career experience in the late 1980s as a young Regional Operations Engineer in NW NSW introduced me to the power of data in energy systems. I developed a basic data acquisition scheme using an IBM PC XT, connecting cables from the operators’ mosaic panel to a small analogue-to-digital card, creating a real-time system. Predating Windows, this setup ran on MS-DOS, with Lotus 123 for data handling and a Turbo Pascal GUI displaying a regional map. Though rudimentary compared to the multi-million dollar Toshiba DAC Systems in larger control centres, it offered a glimpse into the growing role of real-time data and relieved operators of manual logbook entries - a process that later inspired the name NEMLog.

By the mid-90s, through my "tinkering" with data, I was involved in establishing the Interim State Electricity Market (SEM) in NSW, a precursor to the NEM, which officially launched in December 1998. At that time, our little group had the foresight and opportunity to publish basic SEM data in real-time, such as prices, demands, and interconnector flows between NSW and Victoria, while QNI was still under construction. This foundational work eventually led to NEMMCO and, later, AEMO, now providing the extensive, real-time data we rely on today.

Interestingly, when I left the TNSP 8 years ago, the role of a energy “data analyst”, a role I had assumed by default throughout my career or more so towards the end, wasn’t really a recognised or significant function in the energy industry and on some days, I still feel the same thinking from various quarters - it's nice to have but not worth the money and effort to find.

As I said to Daniel "I was sort of a data analyst before there was data analysts." Daniel’s question about the rising importance of data made me realise just how far we've come, and how essential data has become in managing the complexities of the energy transition and how to make the most impact from our efforts. Here are some of my thoughts on this.

What is energy data?

Energy data refers to the vast amount of information collected from our energy systems—whether it's the output of wind and solar farms, consumer demand patterns, market prices, or network performance. This data provides real-time visibility into how energy is generated, stored, transmitted, and consumed. From the smallest rooftop solar installation to large-scale grid operations, every node is a data point feeding into the bigger picture.

Extracting information for better decisions

The value of data lies in how we transform it into actionable insights through our observations. With powerful analytics, we can extract vital information that helps us. A few examples:

• Optimise renewable integration: By predicting generation patterns from wind and solar farms, we can better align supply with demand and minimise curtailment.
• Manage risk: With intermittent renewable generation, data helps manage operational risks by forecasting potential shortfalls or surpluses, improving grid stability.
• Support investment decisions: Reliable data drives better decisions for investors, regulators, and policymakers, helping them understand where and when to invest in new technologies or infrastructure.
• Improve system resilience: Data allows real-time monitoring and adjustments, ensuring that the grid operates reliably despite the inherent variability of renewable energy.

Why has data grown in importance?

Ten years ago, in a one-way energy system where energy flowed from centralised generators to consumers, data wasn’t considered as vital. The system was relatively simple and predictable. But today, with the rapid rise of renewable energy, distributed generation, and battery storage, the grid has become far more complex and dynamic. The shift to two way energy flows - where consumers can also be producers - means that data is now essential to managing this new reality.

With greater complexity comes greater risk, but also greater opportunity. Data provides the insights we need to mitigate those risks, offering clarity amidst the uncertainty of weather patterns, market fluctuations, and shifting demand. However, as Emre Soyer and Robin M Hogarth pointed out in their book The Myth of Experience: Why We Learn the Wrong Lessons, and Ways to Correct Them [3], we must be cautious about assuming that the experience and data we have are always sufficient for the decisions we face. In dynamic and rapidly changing environments, like the energy transition, we often don’t know what we don’t know, or maybe the outcomes of our decisions.

Accessibility and privacy concerns

Another critical issue is accessibility. For data to deliver its full potential, it must be available in open-source, standard formats that enable diverse stakeholders - from policymakers to energy innovators - to extract insights. But with this accessibility, we must also consider privacy concerns. As more granular, consumer-level data is gathered, ensuring data privacy and security will be crucial to maintaining public trust and complying with regulations.

The danger of incomplete information

When we don't have the right information - or worse, when we don't know that we are missing critical data - decisions can be flawed, leading to unexpected outcomes. In the context of Australia’s energy transition, this could mean underestimating risks, overbuilding the wrong infrastructure, or failing to fully integrate renewable sources into the grid. The key is to continually question our assumptions, refine our models, and ensure that the data we rely on is comprehensive and relevant to the decisions at hand.

Why does this matter?

As the energy transition accelerates, the ability to turn data into actionable insights will separate successful transitions from costly missteps. But more importantly, we must ensure we’re not only gathering data but also asking the right questions to avoid blind spots. And as data becomes more accessible, we must strike the right balance between open data for innovation and protecting sensitive information.

Data is no longer just a tool for monitoring, it's a strategic asset for navigating uncertainty, mitigating risk, and enabling better, more optimised decisions. With the right data, we can achieve improved outcomes, such as greater efficiency, lower costs, and enhanced system resilience. This ultimately supports a more reliable and affordable energy system as we move toward a cleaner future.

References:

1. Australia 'wasting' record amounts of renewable energy as share of wind and solar soars By energy reporter Daniel Mercer (Sat 07 Sep 2024) .. https://www.abc.net.au/news/2024-09-08/renewable-energy-wasted-as-australia-greens/104321770

2. Excess renewable energy wasted (Program: AM Broadcast Fri 6 Sep 2024 at 7:00am) .. https://www.abc.net.au/listen/programs/am/excess-renewable-energy-wasted/104318390

3. The Myth of Experience: Why We Learn the Wrong Lessons, and Ways to Correct Them by Emre Soyer and Robin M Hogarth .. https://www.amazon.com.au/dp/1541742052

4. Understanding Learning Environments: Insights from Robin Hogarth by Geoff Eldridge (Linkedin Post) .. https://www.dhirubhai.net/posts/geoff-eldridge-nemlog_kind-and-wicked-learning-environments-activity-7197036676420915200-53cG

#EnergyTransition #DataDriven #RenewableEnergy #Curtailment #NEM #CleanEnergy #AustraliaEnergy #Sustainability #Innovation #RiskManagement #DataInsights #GridResilience #EnergyMarkets #DataPrivacy #OpenData