Modelling as a Science, not a Hype

I have been dealing with models, statistics, and experiments since 2005 when I started my bachelor in Health Psychology. Sure, I received education in Frequentist statistics before that, but from 2005 onwards I did at least two exams on statistical theory and experimental design each year. As my training progressed, I moved towards edvidence-based medicine, epidemiology, and simulation. In conjunction, I learned and dealt with model innovation and risk-communication. All of what I learned during my education and ongoing work-experience I try to distribute through my work and consultancy which are beautiful learning experiences on their own.

Models

Your perception of what a model is most likely depends on the scientific field you are working in. For mathematicians, a model is made up of formulas and numbers, but for a social scientist a (mental) model can be made entirely out of text. What they both share is their desire to discover, describe, and mimic worldly processes. To obtain a better understanding of the world and achieve some level of control.

By using mathematics and statistics, and by relying heavily on content knowledge coming from scientific fields such as biology, physics, and chemistry, scientists can create models to make sense of collected data as well as physical and biological phenomena. Of all the models in the world, weather forecasts have undoubtedly the biggest impact, as everyone is affected by what they predict, or fail to predict. They are also amongst the most chaotic and difficult processes to understand, despite decades of research and a deep integration of physics, mathematics, and a tremendous amount of real-time data.

In the animal sciences, where I now spend most of my time, there is a (fading) separation between mechanistic (biological, top-down) and empirical (statistical, bottom-up) models. Animal scientists can follow special training to become a mechanistic modeler which has led to most animal models also being mechanistic.

This does not mean that animal scientists are not familiar with statistical procedures such as ANOVA’s or regression. It just means that there is a preference to model biological processes via a particular modeling type. In the end, most modelers are familiar with both ‘worlds’, albeit at a different levels of experience.?The availability of applicable machine and deep learning models has given rise to more statistical models being developed, which from time to time drives the discussion of their inherent differences. These differences exist, but I could make a good argument for proclaiming that mechanistic and statistical models have more in common than meets the eye.

For one, both types are models, and both are driven by algorithms that allow them to connect input to create output. ?Algorithms themselves are mathematical formulas that can help models find solutions, and most models employ multiple algorithms in conjunction. Although some have a closed-form expression, like the differential equations used in mechanistic models (like the Covid-19 SEIR models), others function by iteratively looking for the most optimal solution. These grid search algorithms often form the basis of empirical models of which Monte Carlo simulation is great example. To make the connection to machine and deep learning easier, we will label ‘empirical’ models as ‘statistical’ models from now on. Also, here, the distinction is more man-made than of actual importance. In the end, it is all semantics.

?It is all semantics

In the past decade, there has been a sharp increase in distinct words that describe the use of statistical models on large datasets: ‘big data’, ‘data science’, ‘machine learning’, ‘deep learning’, and ‘artificial intelligence’. In conjunction, older job titles such as ‘(data) analyst’ or ‘(bio)statistician’ have been replaced by ‘data scientist’ or ‘machine learning engineer’. Data is the new gold, and artificial intelligence (AI) will replace millions of jobs. ?

A graphically appealing way to highlight the emergence of data science, and the extensive marketing around machine and deep learning models, is to take a close look at the Gardner Hype Cycle. The Hype Cycle tracks technological developments across time, and there is a specific cycle for developments in the field of AI. It has five distinct phases: ‘Technology Trigger’, ‘Peak of Inflated Expectations’, ‘Through of Disillusionment’, ‘Slope of Enlightenment’, and ‘Plateau of Productivity’. Not every technology makes it to the productivity phase and technologies can be placed back in the curve. At the time of writing, both machine learning and deep learning have just passed the ‘Peak of Inflated Expectations’ moving straight ahead for the ‘Through of Disillusionment’. For most data scientist working in commercial settings, this is not a surprise.

The importance of the Hype Cycle can be discovered by tracking each development to its origin. The algorithms that feed deep learning models (i.e., neural networks), for instance, have been around for at least fifty years. Similar cases can be made for many of the machine learning models used today. A linear regression model that estimates its parameters via Maximum Likelihood, and not Ordinary Least Squares, is a model that acquires its solution through an iterative process. It therefore fits the criteria of a machine learning model, but the model itself is much older than the term ‘machine learning’. The changes in naming and framing of old concepts is what brought the field of modeling a lot of unnecessary confusion. Unfortunately, it seems that with every new edition of the Hype Cycle, more and more new old terms are being introduced.

Splitting technological developments into distinct fields can bring added focus, but at the cost of losing the overall common denominator of what it means to build models in the first place. Every model needs a reason to exist, and a human to build, evaluate, and deploy it. Having changing names for the same type of models does not help the true challenges of modelling, and deviates energy away from developments in the field of explaining and predicting worldly processes. Models needs to be accepted to be deployed and generate a return of investment, and acceptance comes from models that can be trusted to explain and / or predict.

?The scientific cycle

The scientific cycle, or ‘scientific method’ or ‘research cycle’, describes a circular process in which a research question leads to a hypothesis, which leads to an experiment, which leads to an analysis, and from which then a conclusion can be derived. Since the endeavor of answering questions often leads to more questions than answers science is very circular indeed.

In the field of Bayesian statistics, models play a pivotal role in connecting new information to previous knowledge. This new ‘understanding of the world’ is labeled ‘posterior knowledge’. Mathematically, it is literally the product of previous knowledge (the prior) and contemporary data (the likelihood). This means that the posterior is a weighted combination of old and new information, which basically means that new information needs to be placed in context. For those of us who have forgotten how that looks like in practice I suggest you observe children at play. They are Olympic champions when it comes to Bayesian inference. Adults would be too if they would stop trying to rule out important new information in favor of protecting a safe old paradigm.?

The power of Bayesian analysis is not just its mathematical description of learning and discovery, but rather its excellent framework for integrating knowledge, models, and modelers into actionable cycles. Contrary to the Frequentist approach of accepting or rejecting a null-hypothesis, Bayesian inference clearly supports uncertainty and the fluidity of evidence. To ‘Bayesians’, the parameter estimates of a model, or the model itself, have an expiration date that ends when a new dataset emerges. In some fields, the renewal of information will happen at a very rapid pace which gives machine learning models their intuitive appeal. Being masters of comparison, machine learning models iteratively compare the fit of new information with established parameters to assess their relevance.

Weather models, for instance, have been augmented years ago by including machine learning models that are able to deal with the chaotic cross-correlations of weather patterns.

This is also why most weather forecasting models are ensemble models – if anything is missed by the first model, some of the other 49 models will pick it up. If a single model places an extreme forecast, the other models can be used to check its validity and ultimately present a weighted and more balanced prediction. The entire process will happen on a continuous base.

To build a model that can continuously learn from the data it is being fed requires a tremendous amount of work on building both the model(s) and the data pipelines that feed it. It also requires a tremendous amount of research driven by many runs through the scientific cycle. Something no modeler can achieve by himself in any reasonable amount of time. Therefore, structure and transparency are so important.

Nowadays, model creation has been made almost too easy via specific libraries that help you deploy a pre-trained machine (deep) learning model in a matter of minutes. Only to find that either the predictions make no sense, or you are unable to explain the model to yourself, let alone others. Furthermore, the model has no commercial appeal since you do not know how to sell or deploy it. All you did was take a model that is openly available (which is a good thing) and use it on a dataset for which you believed it had some merit.

Most modelers would scoff at the idea of believing they would place themselves in such a situation, but I have personally witnessed more than one example in which (novice) modelers hung themselves by employing that which they did not fully understand. In fact, I may even share in these experiences, and my failures do not limit to statistical models alone. Nowadays, all forms of dynamic systems modeling, from agent-based models to dynamic networks for infectious disease epidemiology, can be found in some pre-build library. As a result, model building has never been more easy and by extension never been more dangerous. Not because the predictions do not make sense, but because less people are able to explain why the model is able to predict as it does in the first place.?As a result, the true teachings follow much later than the first steps made.

A return to the scientific cycle is warranted if we are to increase the uptake of already available models, and the development of new ones. Not only because this would force researchers to become more transparent in their work, via publications and open-source sharing, but also because it would place an already-known structure on top of development. Whereas the publication of models via traditional media can be slow and incomplete, openly available repositories like GitHub are not. Via GitHub, researchers can share contemporary work and openly invite other researchers to augment their code. Version control is automatically deployed to make sure individual contributions can be tracked, and the original work is not lost.

As a result, there has never been a better time to share your work as over half of new technological developments are now made available via open source leading to great opportunities for any modeler willing to lay out his or her cards. ?

The virtue of any scientist!