The DWMP blog – 29. “All models are wrong but some of them are useful”

The title of this episode of the blog is a quotation that I have often used when talking about urban drainage models, in fact I used it in Episode 4.?I was prodded into blogging about model accuracy by a research paper that took this quote as its title.?(Pedersen 2022 https://lnkd.in/eNu_Pe3e).?The paper presented an innovative statistical method of comparing model results with long term water level measurements in sewers to define the “accuracy” of the model.?I compared this with earlier work on defining model accuracy: the CIWEM Urban Drainage Group code of practice, model data scoring tools developed by some UK water companies and an UKWIR project (sadly never published) on moving away from short term flow surveys as a way of defining model accuracy.

It should be obvious from the blog title that I think that we need a way to defining model usefulness.?Accuracy in reproducing any particular flow event is less significant.?

Pedersen set out a useful characterisation of the sources of inaccuracy in urban drainage models, where inaccuracy is the disagreement between model and measured conditions.?See the diagram below reproduced from the paper.

To think about usefulness, we need to consider that the measured data might also be inaccurate.?I therefore sketched the diagram below that shows the components that lead to a widening band of uncertainty in both model and measured data.?The actual position of the results within that band is unknown for any individual event but they are unlikely to meet in the middle.

How can this structure help us to not just measure the usefulness or accuracy of a model but also inform what we need to do to improve its usefulness.?To do this we need to consider the significance of each of the factors causing uncertainty and how we can reduce that.?I base my ideas below on UK practice, the balance may be different in other countries.

Model uncertainty

• Asset data – errors in sewer line and level and pump capacities
• Catchment data – uncertainty in connectivity of impermeable surfaces
• Data input errors – incorrectly transposing data to the model
• Model concept – the equations used in the model
• Model errors – coding errors and computational instabilities
• Antecedent data – uncertainty in defining initial conditions

Asset data

Our asset data is not as good as we think it is.?There is a general belief that sewer survey data is the gold standard of accuracy.?It isn’t, it is frequently wrong.?Errors here often show as errors in water levels rather than flow rates.?My test for this type of error is to look at a sewer long section and imagine a junior design engineer drawing it out and asking a senior engineer to approve it.?If that isn’t plausible then it is probably wrong, even if it has just been surveyed.?We need to put more effort into reviewing sewer data as we build our models.?Automated scripting tools can help with this.

Catchment data

In many catchments with separate or partially separate foul and surface water sewers there is considerable uncertainty over which surfaces drain to which system.?This tends to be the first area of adjustment when comparing modelled and measured results.?This remains a challenging area for improving the initial input data and will remain the most important use of measured flow data.

Model concept

Conceptual models of sewer hydraulics are robust (although there are some issues around representing pump performance).?However, there is a wide range of conceptual models of rainfall runoff, none of which truly represent what is happening.?There are significant issues with representing slow infiltration response to rainfall and with representing the limits of system inlet capacity.?These lead to inaccuracy for large rainfall events, which is when we most need accurate model results.?To avoid this we need to test the conceptual response of the models to a wide range of conditions to reduce these inaccuracies before we address the accuracy of the models for individual rainfall events.

Antecedent data

The definition of starting conditions for the model such as catchment wetness, soil moisture and storage tank levels can cause significant inaccuracy particularly for small events, but this can be reduced with continuous simulation of a string of events with an appropriate conceptual model.

Measured uncertainty

• Asset performance – local, short-term issues such as partial blockage
• Rain sensor density – the spatial density of rainfall measurements
• Rain sensor accuracy – the inherent accuracy of rainfall measurements
• Rain sensor errors – rain sensor faults leading to no data or corrupt data
• Monitor location – how many flow monitors and where they are located
• Monitor type – what is measured and how it is measured
• Monitor errors – flow monitor faults (or operational problems) leading to no data or corrupt data

Rain sensors

I have intentionally used the phrase “rain sensor” as I include the use of radar data as well as conventional terrestrial rain gauges.?The two types have different characteristics with radar providing high density measurements but reduced accuracy and rain gauges higher accuracy but lower density.?Combining the two sources would be the best option.

Some might argue that inability to accurately represent the spatial variation of rainfall is a model inaccuracy, but I disagree.?Inaccuracy in representing the spatial pattern of rainfall in any one event does not demonstrate that the model is inaccurate in other events or that it is not useful for planning or design of future conditions.

Monitor location and type

There is a similar balance between spatial resolution and accuracy for flow monitors, but limited by cost rather than technology.?A high density of gauges will require the use of cheaper gauges such as level sensors that are less accurate in understanding the mechanisms in the sewers.

Pedersen’s work is based on measuring water level.?This is valuable as level is what drives flooding and overflow but it does not give an accurate understanding of the mechanism causing the measured levels.?In the UK we found the need to add velocity measurements in the 1980s to give us a better understanding of the flow conditions.?Was this the correct choice?

To select the monitor location and type it is useful to consider which model inaccuracies we are trying to target.?If we target asset data errors through data checking and runoff model conceptual errors through testing robustness of the model to a wide range of events, then the main issue to be addressed is connectivity of runoff surfaces.?For this I think that we ideally need to measure flow rather than just level and we need locations to cover each landuse and era of construction rather than necessarily each geographical area.?How valid would this approach be?

Conclusions

We need to focus on model usefulness rather than on the accuracy with which model results match inaccurately measured flows caused by uncertain rainfall on a catchment of unknown condition.?We should aim to eliminate asset data errors and model concept errors before we start to compare the model with measured flows and levels.?We can then concentrate on the connectivity of impermeable surfaces which is the thing most difficult to measure directly.