Continuous Model Build and Calibration - Why Water Models Should Build Themselves

Instead of building models, modellers should dedicate their time to supporting operational and planning staff in understanding and using the models

With live modelling becoming more prevalent in the water industry, operational and planning staff are now expecting immediate feedback on changing conditions in the network. Gone are the days when a modelling group could expect a calibrated model from years ago to meet all the requirements of their water company.

Let's remove the engineers from the model build process and allow the models to build and validate themselves. Having a robust validation process in place before a model build would ensure only high-quality input data exists in all corporate systems and would eliminate the need for a manual approach to building models.

Instead of building models, modellers should dedicate their time to supporting operational and planning staff in understanding and using the models. If we give the models directly to operators and assist them in using them, it would lower scepticism among operators and stop the old repeated line between modellers that “operational staff don’t understand modelling principles.”

What’s wrong with the current method?

While there is indeed a level of expertise (and sometimes what feels like artistic talent) needed to make a model work, the bulk of a model build is purely a data cleansing and documenting exercise, with only a fraction of the time dedicated to calibration.

The model build and calibration of a moderately sized water supply area may take anywhere between two to five months. An engineer will spend hundreds of hours on a single model, as they oversee all aspects of the build and calibration, including planning field tests and writing the final documentation.

While proactive maintenance work can be done to keep models current, the process of comparing and updating models is slow, tedious and not as fulfilling as doing a new model build.

It is just easier to start from scratch than to try to understand what has changed in the system.

Water utilities have historically created models for long-term planning purposes. Updating a model with minor works such as pipe renewals or new developments has minimal impact when looking at 25-year master plans. Once enough time has passed and the model no longer accurately represents the network, it is just easier to start from scratch than to try to understand what has changed in the system.

This previous approach of using older models does not work for live models. For them to be most effective, they need to be as close to the current network as possible.

The problem is that typically a few months will have passed between the calibration day and when the modeller finishes the model build and calibration. As a result, the model could be outdated just enough that operational staff lose confidence in it.

So how do we fix this? We manually build the model once and then push it into a system that keeps creating the model over and over. I call this a continuous model build.

Continuous Model Builds

Validating the raw data

In software development, it is typical for source code in large applications to be compiled and tested nightly when multiple people are working on different sections at the same time to ensure that changes by one person do not affect the work of another. Models could be built using this same principle.

Instead of the modeller manually collecting data from different sources and modifying or correcting it to work for the model, we could create a system that automatically validates and provides feedback of the overall quality of the data. If the data is of sufficient quality, it will build the model automatically. However, if the system detects significant issues, the original data owners would be notified and prompted to update the data to enable a successful model build.

The way I see this working in practice is that every night the entire service area for the water utility is built as one interconnected network. The network is then traced downstream of every service reservoir and meter to ensure connectivity and prove the boundaries as drawn in GIS.

The system will highlight any traces that do not match the areas defined by the polygon boundaries. It will also highlight any disconnected assets from the network or those that are missing critical attributes such as material or age.

The system will generate a report each night giving each modelling areas a rating. Zones with scores above a set threshold could build themselves automatically; the remaining zones would require the data owners to fix enough issues to allow the failed zone to create itself the next day.

With the entire service area traced, it would be possible to extract each water systems in one piece from GIS – no more cutting rectangles of the system and spending hours trimming down the model.

Future updates would be quickly passed down from GIS to the model. If a DMA boundary changed, then the GIS team would only need to update the polygons and the relevant valves, and the next day a model would be created with the new network layout.

Automating the build

Most advanced modelling packages now come with the ability to build models and run scenarios programmatically. Demand analysis can also be completed within most modelling packages, eliminating the need for demand analysis sheets that are done by hand.

For example, InfoWorks can use the Ruby programming language, either from within the user interface or with an extension product called InfoWorks Exchange that allows scripts to control all aspects of the modelling software from the command line.

Validation rules within InfoWorks can also be run directly from Ruby at the command line, and the nightly report could include the output of this validator. 

The example video below demonstrates how scripts I recently wrote extract raw data from a GIS server, then filter assets, process fields, split pipes at nodes and expand short links. For more details on this particular script and what it is doing behind the scenes, you can read more on my earlier post linked below.

Keeping the calibration

While pipes and minor fittings such as valves and hydrants can be imported directly from GIS with minimal processing, the same cannot be said for critical assets such as tanks, control valves and pumps.

In general, the modeller sets these by hand during the first model build and calibration to control the dynamic nature of the network by analysing downstream calibration points and advice from local operators.

A database would store the controls from the initial calibration, which could then be reconciled by asset ID and reapplied during each nightly model build.

Generally, two categories of calibration settings exist. The first includes physical actions (e.g., float valve operating ranges, set points for PRVS or pumping stations) and sizes of control objects (e.g., tank volumes). The second includes calibrated actions that mimic previously observed calibration anomalies, such as throttled valves to simulate suspected headloss from point restrictions or modified roughness values on pipes outside standard values.

In addition to having controls, some assets have either minimal or excessive pipework around them. In these instances, it would be beneficial to store a spatial representation of the simplified pipework required for the asset to function, as well as flagging any redundant pipework so that it is not imported.

Validation against live data

Once a model is built and runnable the system pushes it into a live modelling environment, and use of the model can begin. An initial run could be conducted immediately on the previous 24 hours to verify the model against live data stored in SCADA.

The live model will issue alarms if pressure and flow loggers are significantly outside observed data, as this could indicate the physical settings of infrastructure have changed, assumptions about the controls of the network are wrong, or there is an issue to investigate within the system.

These issues can also be flagged and added to the daily report to be investigated by relevant staff.

Maintaining calibration

While SCADA data will provide some indication of the effectiveness of a model, the available sensors in a typical utility's telemetry system would not be sufficient to cover the general rule of thumb of having one pressure logger per 200 properties.

For a moderately sized water supply zone, you may need to deploy anywhere between 100 to 200 pressure loggers. Field tests of this size end up being a logistical nightmare. Modellers are required to organise the hire of loggers, their deployment by field staff and, once they have returned, the manual download and analysis of the data.

Instead of large-scale deployments to recalibrate entire water models, a rolling schedule of calibration at a DMA level would be more efficient. Modellers could use existing low powered data loggers with mobile connections that communicate with the utility’s telemetry system directly. The model build tool would already require a link between telemetry data and the physical infrastructure. It would be simple enough to extend this to include a registry of floating remote sensors.

A DMA requiring recalibration would have remote sensors placed at the required hydrants for calibration, the pressure recorded would transmit from the data logger each day, and the modeller would update the database with the new locations associated with the remote loggers. The next day, the model would be built with the live data automatically associated with the correct hydrant.

As the live modelling software automatically verifies pressure data, the modeller would only need to wait a single day before being told if the DMA is within the required calibration tolerances. If there are any anomalies, then the modeller could further calibrate the model and update the calibration database which would be included in the rebuilt model the next day.

Once the model provides acceptable readings compared to the remote sensors, they can be picked up by field staff and moved to the next scheduled DMA. Deployment and calibration could be staggered over multiple DMAs so that modellers could actively calibrate a zone while field staff moved loggers from recently completed areas. If the field test detects no issues in an area, it could be closed out quickly, while those with significant anomalies could have extra loggers deployed to help identify issues as the field test is happening.

Proving boundaries or finding point restrictions such as partially shut valves could use the same technique and be resolved within a few days.

When can this be done?

The only piece I feel is missing in the process described above is a tool to validate and report on issues before a model build begins. Starting next year, I will develop a piece of software that does just that.

With this software, along with my other tools, I believe any water utility could implement a fully automated model build system with relative ease. There would be a moderate capital cost to fix the initial data; however, the water company could easily justify this cost against the current price of rebuilding models, the added benefit of consistently high-quality corporate data, and access to live models.

I know this is an area of interest for many people in the industry and while there is more and more talk on the subject, I have yet to see an entirely automated model build system. If you have done one or know of one in the works, I would love to hear more about it.

Is there something I have missed? Are you still sceptical? Worried about your job as a modeller or just excited by these new possibilities? Either way, please leave a comment below or share this article!