Transport models: Thinking outside Box

I have always been in a kind of thrall to transport models, but using them in forecasting and planning is my real focus. Transport modelling is important, but I don’t believe that it and forecasting are the same thing. Models don’t forecast. They provide information on how the complex interactions between demography, economy, geography and infrastructure affect travel patterns. When models are used well, they give us the means to form a sensible? view of what the future of transport may look like and plan for it.?So I’ve always been a little puzzled by the George Box quote:

“All models are wrong, but some are useful”.

I understand that he was saying that no mathematical model in any field will ever represent reality exactly. Some models, though, may be close enough to reality to be used. As a statistician, his quote seems to refer to statistical error and it seems quite clear what he refers to. But when we look more closely, his quote is less clear than it seems. When we use transport models, what does it mean to be close enough to reality to be useful? How can we tell if a transport model’s outputs for a future year are wrong??

It may be that the issue is all about semantics and the meanings of the words. However, the words we use are important, especially with complex issues like transport demand and forecasting. Croesus, king of the Lydians, consulted a bunch of oracles in what must be the earliest example of calibration. Even then he misinterpreted the Delphi Oracle’s prediction though? (If Croesus goes to war, he will destroy a great civilization). His own civilization was destroyed, probably because he underestimated its greatness and because of the ambiguity of the prediction.

I have no doubt that the words in this article, which I think I understand fully and have picked as carefully as I can, will be unclear and confusing to many. I hope no civilizations are destroyed.

A transport model’s usefulness

The idea that some models are useful suggests that some are not. I would have thought that models are purposely built to be useful, to do something according to a specification; to answer questions we may have about transport demand. Models are useful almost by definition and the idea of a model not being useful is meaningless. Or perhaps we simply assume that whatever model we are using is one of the useful ones. But a model that isn’t useful isn’t a model, it’s a mathematical curiosity.

The context of the quote suggests to me that the usefulness of a model is based on its calibration. The calibration tells us how closely it comes to reality, with guidelines providing the limits of what reality is. The problem is that there is no way to know whether the model’s forecast is useful or how wrong it is. It could be that the model’s calibration tells us that it’s very useful, or wrong in only a minor way. But if we are interested in the performance of new infrastructure, the model may be too sensitive or not sensitive enough to the new infrastructure. Years later, in retrospect, we may find the model was spectacularly right in all areas except around the new infrastructure. It’s not easy to see whether the model was wrong or not, useful or not.? Obviously, for the purposes of assessing the new infrastructure, the model ended up being wrong. If the model was a planning tool, it would not have been wrong and would have been very useful.?

Models and wrongness

The idea that all models are wrong is troubling because it covers such a wide variety of possibilities.?

It’s not at all clear (at least to me) what “wrong” looks like in a transport model and how this wrongness can be fixed, or even assessed. Perhaps it’s about precision or accuracy or lack of it. We all know that models, especially strategic ones, are based on aggregate information from small samples and diverse sources that are not always mutually consistent.? The assumptions that we make to fill the gaps in the data may be okay for the present and the past, but may not be in the future. We also know that in a strategic model, looking too closely at a local area reveals differences from reality. A strategic model’s estimates of turning volumes? at an intersection or of entries and exits at a station rarely match counts. But these are all about error, not wrongness. There are errors in almost everything humans do and as long as we have some tolerances built into our procedures, they can be accommodated and accounted for. Errors don’t make a transport model wrong. Nevertheless, it seems that? these are what Box was referring to. Strategic models are built to represent broad, high-level facts, so differences in localised areas in a model, or statistical errors in aggregate measures doesn’t? mean that we should think a model is wrong.

Maybe a model’s wrongness lives in uncertainty. The upheaval of lives during the pandemic threw uncertainty about future travel choices into a stark light. We pay a lot more? attention to the unknowns in a future that we now think is more uncertain than we used to think. Perhaps the most common way to tackle uncertainty is to test alternative scenarios that encompass the range of ways that the world may change in the future. I think there’s a second layer to uncertainty, though, that is wrapped up in how well our models represent travel choices in those alternative scenarios. We won’t know how the model performs until we get to the year we forecast for, which is not at all helpful. To deal with both layers of uncertainty, it may not be enough for us to model alternative scenarios, we may need to model scenarios within those scenarios.?

Recently, the modelling of the spread of coronavirus introduced me to the Uncertainty Principle of Social Science, or the curse of dimensionality. I had heard about it before but never paid much attention to it. The principle describes how the number of independent factors (called dimensions) in a model limits the chance that a model can closely represent reality. It is not about imprecise measurements or statistical errors in data. It is a real limitation, recognised and dealt with by data scientists, for example, who try to reduce its impact? by increasing sample sizes.??The clearest non-technical? explanation I have read on this topic is an article in Scientific American, which explains in brief as follows:

“We want to make our models more descriptive by adding more interacting elements, but the curse of dimensionality all but guarantees that if you try to fit a model with a large number of parameters to data, your fit won't be close. We can get good estimates of the effect of a public health campaign, for example, in a broad context with few details, or we can get an imprecise estimate in a focused and detailed setting, but a high level of detail and a precise estimate for all those parameters is near impossible.”??

(https://www.scientificamerican.com/article/the-heisenberg-uncertainty-principle-of-social-science-modeling/ )

I have a strong sense that this is important for transport models, because we use a lot of interacting elements. We could echo the explanation from Scientific American: we can get a good estimate of the future demand for transport in a broad context with a few details, or an imprecise estimate in a focused and detailed setting, but not both.? I confess that I don’t know how significant this is for our models and how to compensate for it. Perhaps it's a topic for research.

Models and morality

There is also a moral meaning to “wrong”, so that “All models are wrong” would have the same context as “All models are evil”. Articles about issues like “toxic” algorithms have appeared on LinkedIn and elsewhere, but models are no more than tools. To call a model evil is the exact equivalent? of calling a hammer evil if it hits the nail on the thumb rather than on the head.?

We have a poor record at forecasting toll road patronage. Transport demand models were used in testing patronage of toll roads in the late 1990s and early 2000s and in every case over-estimated toll road demand.? Companies failed as a result. Consultants involved in the toll road studies were charged in litigation with:

• Misleading and? deceptive conduct
• Negligent misstatements
• Engaging in a fraudulent scheme? to manipulate patronage forecasts?

Respected academics and practitioners accused planners and modellers of? engaging in deceit. Jan Flyvberg wrote that:

“... planners lie with numbers. Planners on the dark side are busy, not with getting forecasts right and following an ethical path, but with getting projects funded and built. The most effective planner is sometimes the one who can cloak advocacy in the guise of scientific or technical rationality.”

Robert Bain warned investors to beware the deceit of planners:

“ To knowingly inflate traffic and revenue projections is an act of deception – but it is not alone in that regard. Investors reviewing toll road studies should remain alert to other potential acts of deceit.”

In 2015, the Bureau? of Infrastructure, Transport and Regional Economics (BITRE) hosted a symposium that investigated all aspects of toll road modelling and demand? forecasting. The symposium identified the usual faults in transport demand models: lack of data, poor quality of data, lack of detail in models that were not sophisticated enough. For me, the most telling item in the BITRE symposium was a submission from a prominent modelling expert with a wealth of modelling experience - an uber-modeller -? which stated in part:

“The real issue here is that if a developer wants to take an optimistic view of the future and ask his traffic advisor to prepare forecasts on the basis of these optimistic assumptions, it is not the fault of the advisor that the forecasts are ‘high’.”

I don’t subscribe to this view, but perhaps the real issue is that if this approach is taken, potential investors should be informed about it in the prospectus. Mostly, I think that the BITRE symposium and the academics missed the fact that models were not really used to forecast the demand on the toll roads. The demand was set by bankers to make a commercial case with their financial models. The uber-modellers then “guided” the modellers in ways to make the transport models’ toll road volumes match the bankers’. Nothing was morally wrong about the models.? They were used wrongly.?

Maybe Box’s quote should be

“All models are useful. We may use them wrongly”

What to do about it

The BITRE symposium spurred the development of reference cases and guidelines for models. Reference cases provide a standard comparison against which a project can be assessed. They reduce the opportunity for a base case to be artificially worsened to exaggerate the benefits of the project. Guidelines are detailed specifications of the construct, contents and performance of the model. These are aimed at, again, reducing the potential for exaggeration of the benefits of a project by limiting the sensitivity of the model to researched and documented ranges.

I'm sure both are effective. Perhaps, thought, we should reclaim our historical roots. When we forecast transport demand, we are engaging in exactly the same sort of activity as the sages and oracles of the past. They consulted the entrails of animals and birds,? spilling the unfortunate animal’s intestines for the interpretation of a sage or seer. Or, a little less gross, at oracles like? the one at Delphi, predictions were delivered as a mad rant of a naive woman stoned on some sort of narcotic. A council of educated priests interpreted her words and turned them into predictions. The Oracle took some advantage of diversity, being a mix of the unschooled and the educated, the inebriated and the sober.?

Whether we are sages, seers or priests, models are the entrails that we examine or the mad ravings that we need to interpret. I think that we could provide a better set of forecasts if we use more than one model (maybe entrails and oracle). Even better, an independent panel of planners who deliberate on the models and their interpretations would probably build a more robust view of the future than a set of runs from a single model.?

To Finish

Planning our future is more important than ever.? I think we need to broaden our modelling processes and to improve our interpretation of their outputs. Models don’t predict the future for us and they never will. It’s up to planners to interpret what models tell us. There is a strong temptation to add complexity to our models, with extra feedback loops,finer zones,? and a wider range of factors. It feels like we’re making the models smarter and that they tell us more about the future in more detail. It’s an illusion, a spurious accuracy where accuracy doesn’t exist. However, these models are not wrong in any sense of the word “wrong”. The wrongness in a forecast stands with anyone who uses the model to make forecasts without understanding its language, whims and fancies.

Overall, I think Box’s quote is about different kinds of models, probably not ones used for forecasting and not transport demand models. It would be ironic if the quote was not useful for transport modellers. Time to model outside the Box (see what I did there?)

(More details on modelling and the Delphi Oracle can be found at https://kwelamanshebeen.id.au/oracles_and_models.ppsx/ or with an internet search).