Reflections on Lubarsky’s Law of Cybernetic Entomology

As modellers, we often have to deal with recalcitrant models that resist validation. One of the reasons for this is that models may contain errors or bugs that are yet to be uncovered and corrected. Enter Lubarsky and his Law of Cybernetic Entomology that simply stated says: “There is always one more bug”. This appears to be true in the case of most if not all transport models.

Transport models generally incorporate a well-developed software package like VISUM, EMME or CUBE that have been tested many times; however, it is not unknown that a particular release contains bugs that are only identified when used in earnest. Moreover, a model usually involves several thousand items of data from surveys, maps and sensors, several hundred parameters (some estimated and some borrowed from other sources), and well over thousand lines of scrip (macros/code) some of it specifically written for the project, other bits a legacy from earlier models. The opportunities for confirming Lubarsky’ Law are practically unlimited!

Errors in transport models

I have probably supervised and reviewed more that my fair share of models in different parts of the world. I have encountered numerous errors and bugs, most of them human errors. These are some real-life examples I have faced:

• Input data errors. Some are simply mistakes: the street direction coded the wrong way even if extracted from a normally reliable (but slightly out of date) source. Some network data can be coded in slightly different ways by different modellers and inconsistences arise if the links are critical. For example, two-lane streets becoming a single wide lane before a merge; bus lanes that also accept motorcycles and taxis but are interrupted a few metres before the junction stopline.
• Physiological errors; the correct formula was meant to be copied to all cells in a large excel model but a distraction (tiredness?) meant that the finger on the mouse was lift inadvertently half the way through so only some of the cells were changed; it had a hidden but important effect in the estimation of future demand.
• Technical errors, often an overuse of an attractive model like Logit without fully understanding its limitations and the fact it always accepts a share of apparently “irrational behaviour”; this could be reasonable for mode choice but not for the choice between a tolled and a free route where logit will allow a proportion of trips to pay the toll for no time saving or even a time loss (there are good ways to avoid this).
• Script coding errors. Python and script languages are not particularly good at communicating what is intended and programmers are not always explicit about intentions; a single bracket displaced by one character can mean gross under or over-estimation of future demand.
• Interpretation errors. All models need good quality interpretation of outputs from somebody who understand the assumptions and the inevitable uncertainty of any future projection. That the “model says so” is a certain indicator of trouble ahead.

Debugging transport models

Modelling errors can be very costly. Even if we humbly accept that there may still be another bug we should try to eliminate all errors during the development and implementation of a model. Some general guidelines are obvious:

• Never assume that input data, whoever collected it, is reliable and error free; check it and double check it.
• Even if there are no obvious errors the data may still be wrong, for example as a result of the way the questions were posed in a Stated Preference survey.
• Always get the model reviewed at different stages by somebody not involved in its development; this could be an internal audit but for an important project it is better to complement it with an external review; allow enough time and budget for this review.
• Get all data normalized to the Base Year for calibration and validation; in other words, do not mix data, for example traffic counts, from different years without adjustments for growth and possible network changes.
• Avoid spurious precision and detail as they are potential sources of error and make interpretation more difficult.
• Sense check everything!

Other error avoidance techniques are more specific to certain aspects of the model, for example:

Networks are often a source of error, even when extracted from an apparently reliable source; they may be out of date or for a date different from that of, say, traffic counts and travel times. This is particularly important for public transport networks. I have found it useful to load a unitary matrix (with a one in each cell) to quickly identify problem areas. Building minimum cost trees will also help identify anomalous routes. Both techniques also help in reviewing the centroid connectors and their influence in route choice.

Trip Matrices usually require a lot of attention to refine them during calibration. Techniques to avoid distorting observations are well known like checking the Trip Length Distribution before and after any significant adjustment. Make sure the “observed” part of any trip matrix is really reliable and not just a feature of sampling.

Mode Choice. Beware of complex nested (logit) choice formulations with many levels and modes. They may be estimated after some effort but may also produce pathological results in forecasting if the nests and parameters are inconsistent with the underpinning theory (e.g. Random Utility Theory). Testing the proposed model on a simple spreadsheet where it can be exercised beyond its comfort zone may help avoiding future trouble.

Assignment. Using SUE (Stochastic User Equilibrium) models is sometimes helpful in dealing with sparse networks and low levels of congestion. However, its results are more difficult to interpret and may hide many network and matrix sins.

Growth. Given unavoidable future uncertainties one cannot get this right as one shot. Checking total traffic growth against simpler calculations is essential as a sense check. Recognise uncertainties and use scenario modelling to test the resilience of plans and schemes. Future trip matrices and networks always have a lot of noise as the future will not toe the line of our projections.

Reference forecast, that is running the model for future years without any change in the network and public transport supply; this will push the model away from the comfort zone of the base year and show where unreasonable results crop up helping to separate the impact of growth from errors. This is a critical test before any real forecasting on a “do minimum” and “do something” cases.

Sensitivity Tests are critical to check that the model performs reasonably beyond the comfort zone of the base year. Transport models are not linear so testing one variable at a time may be misleading.

Lubarsky reminds us that we can never be certain our model is error-free. Of course, we should strive to detect and correct all the errors we can find. Clients and modellers should allocate sufficient time and budget to this task. We should also be modest in any statements we make interpreting model outputs.

In my experience over-complexity and excessive granularity (to increase realism) not only make debugging the model more difficult but also generate an unreasonable confidence in the results, inconsistent with its limitations and future uncertainties. This is the realm of Zadeh’s Principle of Incompatibility that I hope to discuss in a subsequent post.