CAIC Weeknotes - Week 3

Jon (Product Lead):

Week 3 at the Centre for AI & Climate has been all about focus. One thing for sure is there are a lot of energy-system-data related problems out there. Our issue has been figuring out which one to work on.

Everytime we thought we were narrowing down, we’d realise our chosen niche was actually a lot bigger than we originally thought, meaning more refinement was needed.

Eventually we had to put our stake in the ground and decide where to start. The reality is the first thing we decide to do will almost certainly be wrong, or at least insufficient in a significant way. But the only way to figure that out is to put something out into the world and get feedback.

This feedback will help us understand what’s missing, and provide us with the opportunity to iterate rapidly and eventually delight our users with something game-changing.

With that in mind we have set ourselves a challenge (well two actually, a challenge each). I would try to put a value proposition down on paper, that was compelling and resonated with our potential users. Steve would try to build a functional prototype that goes someway to backing up our value proposition.

Both are really starting to take shape and are coming together nicely. We’re all getting together in Exeter on Monday to review progress and make some key decisions.?

The next step is to put these in front of users and get feedback, which we will do over the next couple of weeks.

That’s all from me. Next week I hope to have a preview of our proposition and prototype to share!

Jon

Steve (Engineering Lead):

This week has been much more focussed for me. After taking stock of our initial shortlist of focus areas, we decided I should start digging into some existing data to get a better feel for it and experience the challenges first hand.

The first dataset I’ve looked at in more detail is Low Voltage Feeder level aggregated smart meter readings. Several UK Distribution Network Operators (DNOs) have made this data available this year for the first time and lots of folks have told how potentially useful it is.

What have we learnt from a few days of looking at it?

• It’s big! Already there are several hundred Gigabytes of files available to download from different DNO portals, and the combined dataset amounts to hundreds of millions of readings per month.
• It’s disparate. We tried to stitch together data from three DNOs, but to do so we had to download hundred of individual files from different web portals, all of which offer different interfaces and options
• On the plus side, it’s surprisingly uniform - give or take some capitalisation and column differences, each DNO has used basically the same schema which makes combining it relatively straightforward.
• However, it does look like there are various data quality issues that we would want to dig into further before giving a full appraisal: missing identifiers, locations and readings in tens of thousands of rows.

Doing this has definitely shown us how even small bits of friction add up to a lot of time spent when trying to work with this data. The example screenshot below took all of 30 mins to produce, once we had cleaned, combined and massaged the data a bit, but that work took days!

Steve