Pitfalls in Mathematical Modeling for Pharma R&D

By Arijit Chakravarty and Madison Stoddard?

The past two decades have seen the utility of mathematical modeling in drug R&D grow in leaps and bounds. There are many different aspects of drug discovery and development that can and should be supported by modeling. Indeed, if a decision is on the critical path, our belief is that it is almost always better answered with a carefully constructed mathematical model, based on the right data.?

But every tool has its limitations, and mathematical models are no different. It’s an undeniable fact that at least some models, to misquote George Box, are “worse than useless”.??

So, what makes a bad model? To answer that, let’s first ask the question- what is a model good for? Every mathematical model is, at its heart, an abstraction of the system that is focused on specific behaviors. The act of abstraction, driven by its use case, is also what defines a model’s utility.?

In this respect, mathematical models are very much like maps. The satirist Lewis Carroll made one of his fictional characters make this fundamental epistemological point, back in the 19th century:??

“We actually made a map of the country, on the scale of a mile to the mile!"...It has never been spread out, yet, ...the farmers objected: they said it would cover the whole country, and shut out the sunlight! So, we now use the country itself, as its own map, and I assure you it does nearly as well."?

Thus, the need to abstract away certain details and zero in on others is a critical part of the process of model-building. Models must be built with a use case in mind, otherwise they – like the fictional map in Lewis Carroll’s story – can end up shutting out the sunlight.??

But that’s not the only way that model-building can go sideways. Let’s dig into the what-not-to-do list, shall we??

?

Warning Signs of a Poorly Built Model?

The Model’s Purpose is Unclear?

One of the biggest mistakes is starting a modeling project without a clear understanding of its purpose. Models are tools designed to answer specific questions. Trying to create a "platform model" that can handle any possible system behavior is an invitation to disaster. Such models become overly complex, time-consuming, and expensive, often leading to irrelevant or redundant parameters. Define the model's use case before you start and focus your resources on the parameters and behaviors that matter most.?

When a model lacks a clear purpose, it often results in wasted resources. The model may end up being too generalized, trying to cover every possible variable, which dilutes its effectiveness. For example, in drug development, a model designed to predict the pharmacokinetics (PK) of a drug at its target site in humans should focus specifically on relevant PK parameters rather than attempting to cover extraneous compartments and processes. You don’t need to know the gut permeability of a drug that will be dosed intravenously, and you probably don’t need to know the kidney concentrations of a drug that is metabolized in the liver and active in the brain.?

To avoid this pitfall, start by clearly defining the key questions your model needs to answer. Is the goal to predict drug concentrations or target occupancy at the active site? To estimate the optimal dosing regimen? To understand potential side effects? By identifying the model’s specific purpose, you can tailor your model to address those questions effectively.?

?

The Model is Overfitted?

Overfitting is a common issue where a model is too closely tailored to the specific dataset it was trained on. This happens when a model has too many free parameters, making it flexible enough to fit noise instead of the actual signal. An overfitted model performs well on training data but fails to predict new data.??

The danger of overfitting lies in its deceptive nature. An overfitted model appears to perform exceptionally well because it captures all the idiosyncrasies of the training data, including the noise. However, this leads to poor generalization when the model is used to make predictions – often a key aspect of model performance in decision-making. In drug development, this can result in inaccurate predictions of drug efficacy or safety that can expose a program to risk.?

To prevent overfitting, use techniques like cross-validation and be mindful of the number of free parameters in your model. Remember the saying: “With four parameters I can fit an elephant. With five I can make him wiggle his trunk.” To mitigate overparameterization, it's essential to strike a balance between model complexity and the amount of available data. It’s valuable to build the model up iteratively and in a stepwise fashion, evaluating it for parsimony after each parameter addition. Metrics like the Akaike Information Criterion or Bayesian Information Criterion can provide a readout on the tradeoff between overfitting and improving the objective function. Techniques like cross-validation, where the data is split into training and validation sets, can help assess the model's performance on unseen data. By carefully managing the number of free parameters and employing these techniques, you can build more robust models that provide useful predictions.?

??

Parameter Values Were Found by Dumpster Diving?

Garbage in, garbage out. Often, large models can become very data-hungry. It’s not unusual to see systems biology or QSP (Quantitative Systems Pharmacology) models with scores or even hundreds of parameters. Where does all that data come from???

The quality of a model directly depends on the quality of the data it's built on. Large, complex models often require extensive data to inform all the parameter estimates, but not all data is created equal. Failing to understand the experimental methods carefully when extracting parameter estimates from papers or in-house data is a common reason for model failure.?

?

It seems odd to have to spell this out, but if you’re deriving a parameter value for a system property in a model from the literature, read the paper! Most importantly, read the methods and make sure that what was measured experimentally corresponds to what you’re modeling. If you’re saying to yourself “there’s no way I’m going to be able to read the paper for each parameter I’m using in the model, there’s just too many parameters for that”, congratulations! You’ve found the problem.??

The issue of “dumpster diving” is made worse by the trendy expedient of using large language models (LLMs) to extract parameters from the literature, as these models can misinterpret complex scientific information. For example, an LLM may miss that an extracted parameter value only applies to a certain subgroup or was derived from an assay that is not translationally valid. Again, if you need an LLM to read your papers for you and derive parameters, you are using way too many parameters. Consider splitting up the problem into more than one model and focusing your models on the specific scientific questions that are being asked.??

Data generated in-house can have its own downsides. Many in vitro studies aren't reproducible, and relying on such data can propagate errors through your model. Even something as simple as a PK assay requires rigorous validation to ensure reliable performance. For example, the concentrations you expect to observe in your PK study should fall within the assay’s range of quantification. If you wind up with all BLOQ (below limit of quantitation) values, fitting will be impossible. An overly noisy assay will also complicate fitting. It goes without saying that a complex model with large numbers of poorly estimated parameters will propagate error to the point where it is useless.??

To avoid the pitfall of inadequate, incomplete, or irrelevant data, first tailor your model’s data demands to what is necessary and realistic. Once you’ve narrowed the model’s scope to what is actually useful, prioritize high-quality, reproducible data to inform those parameter estimates. Validate in vitro findings with in vivo experiments when possible. When using literature or in-house data, make sure you understand exactly what was measured. A lack of attention to detail when selecting parameter values for models is a major reason for model failure.?

?

The Datasets Used are Ill-suited to the Question?

In some cases, trouble arises from using datasets that do not address the question. In the same way that a model’s structure should be fit-for-purpose, the dataset used for modeling must be suitable for the question at hand.??

For instance, building a model for a chronic disease using short-term data or predicting long-term pharmacokinetics (PK) with only a few timepoints near the timing of peak concentration is problematic. If you don’t sample the terminal phase in a PK study, you won’t get good estimates of the terminal half-life.???

In general, models are best used for interpolation rather than extrapolation. Ensure your data covers the necessary scope and duration to answer the questions your model is designed to address. For chronic disease models, gather longitudinal data that captures disease progression and treatment effects over time. For pharmacokinetic models, collect data at multiple timepoints to accurately characterize the drug's absorption, distribution, metabolism, and excretion. Techniques like D-optimal design can guide timepoint selection when designing PK experiments. By using adequate and relevant datasets, you can improve the accuracy and reliability of your models.?

?

The Assumptions are Wrong?

This is a particularly insidious failure model of model development. Assumptions are the backbone of any model, and incorrect ones can lead to significant errors. For example, if you’re modeling an oncogenic pathway and assume that pathway inhibition leads to apoptosis, you may be able to make a model fit existing data well. However, in real life (both in tissue culture and in patient tumors) apoptosis is rarely the mechanism by which oncogene-targeting therapies exercise their effect. (Cancer cells respond to these therapies via a complex and variable mix of different types of cell death and terminal cell-cycle arrest). Failing to address this reality (for example, using a biomarker for apoptosis as a readout of cell viability) can lead to the wrong inferences. A model that relies on the wrong mechanistic assumptions will make poor predictions.??

To avoid this pitfall, regularly validate your assumptions against empirical data. As new information becomes available, update your model to reflect the latest understanding. Use sensitivity analysis to determine how changes in assumptions impact model predictions. By rigorously validating and revising assumptions, you can build more accurate and reliable models.?

?

The Starting Conditions for Model Fitting Were Chosen Incorrectly?

In simple terms, a model finds its “fit” by minimizing an objective function through systematic adjustment of the model’s free parameters. Fitting algorithms vary in precisely how they systematize this process of parameter optimization, but most rely on a set of starting parameter estimates. The algorithm then adjusts the parameter estimates from this starting point such that the objective function decreases until no further improvements are possible, at which point the fit converges.?

This paradigm can sometimes be challenged by complex models with multiple parameters. In some cases, the objective function may have more than one local minimum that will result in apparent convergence. The minimum that is reached – and whether or not that minimum is the global minimum – will depend on the accuracy of the parameter starting estimates.?

In some cases, such as with PK models, choosing good starting estimates is relatively easy- there are formulas for estimating the clearance or volume of distribution of a drug. In other cases, there may be pre-existing experimental data that can be used to anchor starting estimates for certain parameters. If certain parameters have limited information to inform estimates, you need to test the model for stability to initial conditions to ensure robustness. It’s also worth considering whether certain parameter estimates are likely to be highly correlated, for example on- and off-rates for target binding in a PK-occupancy model. Eliminating correlated parameters by fixing one of them based on experimental data or reducing them to a single value (such as a KD in this case) can mitigate this source of model instability.??

?

The Model was not Validated?

Model validation is a critical step that is often overlooked. Ideally, validation involves testing the model against independent datasets to ensure it can generalize beyond the data it was trained on. A validated model is more likely to provide reliable predictions and insights because its ability to predict unseen data has been quantified. Without proper validation, a model may perform well on the training data but fail when applied to new, unseen data. In drug development, this can result in inaccurate predictions of drug efficacy or safety.?

To validate a model, the ideal case is to test model predictions against independent datasets not used in the training process. If independent test data is unavailable, cross-validation techniques can also assess model performance on different subsets of data. Metrics like the model’s condition number can provide insight into whether some parameters have much greater impact on goodness-of-fit than others, while individual parameter standard errors reflect how precisely estimated they are (within the assumptions of the fitting algorithm – parameter standard errors may be misleading if the model fit is unstable!)?

In the case of a toy model, where no fitting has occurred, a lot can still be learned from a sensitivity analysis. Sensitivity analysis systematically varies the parameters in a model to determine how these changes impact predictions. Given your level of certainty around the value of each parameter or the plausible range for each parameter’s value, what is the range of possible outcomes? Which parameters most significantly affect outcomes? An analysis like this can reveal the likely range of outcomes and identify system properties worthy of further evaluation or optimization.??

?

Case study: Ex vivo data selection for preclinical-to-clinical projection?

In our work for a small pharmaceutical client, we designed IND-enabling ex vivo studies. The client had initially used a PBMC (Peripheral Blood Mononuclear Cell) assay, which lacks a plasma protein matrix. However, their drug was highly protein-bound. As a result, a considerable scaling factor was required in translating the ex vivo activity to clinical projections. This resulted in significant error propagation in the development of clinical projections. To circumvent this issue, we replaced the PBMC assay with a whole blood assay, in which plasma proteins are intact. This more translatable and reproducible assay allowed more robust model-based clinical projections, supporting confidence in a starting dose in the clinic that was fifteen-fold higher. In order to build the right model, consider its application in the program context. In this case, the solution to the problem was to change the strategy, rather than to find a better way to project a scaling factor for a plasma protein binding estimate.??

?

How to get it right when model-building?

To avoid these pitfalls, follow these best practices in model-building:?

1. Define the model’s purpose: Clearly articulate the questions your model is designed to answer and focus on relevant parameters and data.?
2. Use high-quality data: Ensure the data used is reliable, reproducible, and relevant to the model’s purpose, and make sure you understand what each parameter represents.?
3. Validate assumptions: Regularly revisit and validate your assumptions against empirical data.?
4. Simplify where possible: Avoid overly complex models with too many parameters. Use techniques like regularization and cross-validation to prevent overfitting.?
5. Test model stability: Perform sensitivity analysis to test the model’s robustness against different starting conditions and parameter values.?
6. Validate your model thoroughly: Validate the model against independent datasets and perform rigorous error analysis.?

?

Mathematical modeling is a powerful tool in pharmaceutical R&D, offering a blueprint for navigating the complex landscape of drug development. However, a model that isn’t fit-for-purpose, is poorly validated, or is built on an inadequate dataset can harm project decision-making and impede progress on the critical path. By understanding these pitfalls and sticking to best practices, you and your team can make the most of modeling to speed up drug development, cut costs, and ultimately bring better therapies to patients.?