Weather Data on Data Layer TS

Have you ever had the feeling that the weather predictions change every few hours? You can now check it out for yourself! Data Layer TS is the key to storing and sharing millions of time series while ensuring low latencies and easy access.

The Deutscher Wetterdienst (DWD) offers the output of multiple numerical weather prediction models for free. The most precise one (regarding the spatial and the temporal resolution) is the ICON-D2 model and it is offered for a large area around Germany (> 750k grid points - 2.2km between two points). The data is overwritten every 24 hours and includes predictions for the next 48 hours. Every three hours a new prediction is published.

To use the data for analytics or model-trainings more than just one day of historical data is needed. In the last few months, a small ingestion pipeline continuously loads data from the DWD into one single Data Layer TS instance. Instead of always overwriting the data with the latest forecasts, even the historical forecasts are stored.

This is done through storing the data of each variable/grid point combination in multiple time series instead of just one. The first time series is always overwritten and contains the latest data (age-0h). The second time series contains the forecasts, which were done in the previous prediction run (minimum age of 3h). Since the prediction runs are executed every 3 hours and always contain 48 hours, this strategy results in 16 different time series per variable/grid point. This way the data can later be compared and used for model-training or back-testing.

Most of the time series predictions are calculated for 15-minute time steps, which is why the instance now contains trillions of data points (for 400+ million time series).

In addition to the ingestion pipeline, I created a dashboard (ICON-D2 Explorer), which enables access to all of the stored data. If you want to have API access to the data, feel free to reach out to me.

Thank you 微软 (& Microsoft for Startups ) for making this possible!

Thank you Deutscher Wetterdienst for providing the data and support!

If you want to build something like this yourself, message me for the source code of the ingestion pipeline or help to set it up.

Data Layer TS is exclusively available on the Microsoft Azure Marketplace.