Why Decision Making Requires Probabilities from Predictive Models

In predictive analytics, there's often a debate: should decisions rely on raw probabilities, or are simpler approaches, like error costs for false positives and false negatives, sufficient? While error cost-based decision-making might seem intuitive, it has critical limitations, especially when dealing with dynamic, real-world business scenarios. Let’s explore why probabilities are indispensable, using mathematical insights and practical examples.

The Case for Probabilities

In decision-making, the expected loss or gain is what matters. This expectation can be calculated as:

Similarly, for potential gains, the expected value is:

Expected gains and losses offer a structured framework for evaluating trade-offs across different alternatives. By comparing the expected values of various options, decision-makers can select the choice that maximizes the likelihood of success while minimizing the risk of failure.

Additionally, expected gains and losses facilitate more efficient resource allocation. By calculating the expected value of each option, decision-makers can allocate time, money, and effort to those that yield the highest returns, while avoiding investments that carry low probabilities of success or high costs of failure.

Expected values are also essential in shaping long-term strategic decisions. By considering both the expected gains and potential losses over time, decision-makers can develop strategies that drive sustainable success while accounting for possible setbacks or risks.

Ultimately, by calculating expected gains and losses, businesses, investors, and other decision-makers can make more informed, balanced choices that optimize risk and reward, maximize resource utilization, and align with broader strategic goals.

It’s important to note that without probabilities, accurately calculating these expectations is not possible. Relying on hard classification decisions (such as thresholding predictions) ignores critical probabilistic information, often leading to suboptimal outcomes.

The Pitfall of Error Costs

Some practitioners advocate for decision-making based on the costs of Type I (false positive) and Type II (false negative) errors. For instance, the cost of a false negative might be roughly estimated as the lost revenue from a churned customer. However, these costs are threshold-dependent, meaning they vary depending on the prediction threshold.

To illustrate this, we analyzed the Iranian Churn Dataset from Kaggle, calculating the average false negative customer value for each threshold.

While illustrative, this approach assumes:

1. 100% contact rate: Every customer is reachable.
2. 100% offer acceptance rate: All offers are accepted.
3. Offer cost is zero: No cost is incurred to retain the customer.

In practice, these assumptions rarely, if ever, hold true. Moreover, based on real-world experience, both, the contact rate and the offer acceptance rate are not evenly distributed across the churn propensity. Also, businesses often tailor offers to individual customers, selecting from a range of options.

Ultimately, the actual false negative cost must be adjusted to account for real-world constraints, resulting in a more accurate calculation:

This adjustment requires the use of multiple models, each providing probability predictions for key factors, such as the likelihood of contact and the probability of offer acceptance, considering the value of personalized offers.

The Bottom Line

Decisions based on probabilities allow businesses to incorporate the nuances of real-world scenarios, like varying customer behaviors, operational constraints, and cost structures. In contrast, relying solely on fixed error costs is overly simplistic and ignores the inherent variability and uncertainty of real-world processes.

By using probabilities, you enable not just better decisions but also more robust scenario analysis and resource allocation. After all, when your decision-making aligns with expected outcomes, you’re not just reacting—you’re strategizing.