Leadership’s role in fixing the analytics models NOW and AFTER

COVID-19 has upended life as we knew it, and our new behaviors are wreaking similar havoc on some analytics models that rely on historical data. How can leaders enable teams to get analytics back on track?

The COVID-19 crisis has put a spotlight on the power and potential of analytics and artificial intelligence. We have heard from leaders across industries and geographies about the many ways analytics have enabled them to more effectively handle the challenges presented by these unprecedented times, from supporting and protecting workers to engaging increasingly digital customers and managing fragile supply chains.

But the crisis has also revealed the technology’s Achilles’ heel. One of the most widely used advanced-analytics techniques (machine learning) relies on the principle that patterns and behaviors from the past will likely repeat in the future. Algorithmic models expose these patterns in data and draw on them to predict what will happen (such as whether a particular customer will cancel a service) and even recommend what the business should do (for example, identifying which offer will most likely change a customer’s mind). This was effective until th pandemic transformed the way we live and work. Lockdowns, travel bans, physical distancing, and widespread furloughs have altered how we shop, where we perform our jobs, how—and if—we travel, and more. Even as communities reopen, we’re a far cry from business as usualThese shifts are reflected in the data feeding into some algorithmic models built before the crisis, reducing their predictive powers and, in some cases, sending them completely off the rails. One energy company, for example, struggled with trading decisions when crude-oil prices crossed into negative numbers, because data scientists built existing models on the assumption that barrel prices would never fall below zero.

Even once the next normal emerges, organizations won’t be out of the woods. Some patterns in data captured during the COVID-19 crisis (for example, extraordinarily high demand for hygiene products) will become irrelevant. New lasting patterns, such as higher consumer spending on digital channels, will emerge, invalidating or reducing the predictive power of pre-COVID-19 data as well.

Despite these issues, pre-COVID-19 models have tremendous capacity to give executives important insights to help them navigate the crisis and the next normal—if leaders are prepared to take the steps needed to shore up those models. The challenge here is not simply a technical one for data scientists to solve. While analytics professionals will play an important role, successfully stabilizing critical models will depend equally on efforts from leadership to recalibrate business strategies for the changing landscape, forge new data partnerships, convene interdisciplinary teams with sufficient diversity, and more.

Where should leaders start?

The framework that leaders can use to prioritize their efforts, evaluate and resolve the risks, and ready their organizations for the future

Reassessing business priorities to focus model-assessment efforts

At its most basic level, advanced analytics is designed to improve decision making. As a result, before leaders evaluate which models they can trust today and which require adjustments, they should first reassess which models they actually need during this profound transition.

The economic repercussions of the pandemic have been wide-ranging and world changing. All industries and companies have been affected—albeit in different ways. Telecommunications companies struggled to keep up with dramatically higher broadband demand, while oil and gas companies watched prices shrivel as automobiles and airplanes sat idle. Consumer-packaged-goods companies raced to ramp up production of essentials, such as hygiene products, while clothing retailers shut stores down.

While this quick-shifting environment affected the accuracy of models built before the crisis, it also led to the need of wholly new analytics capabilities to help leaders tackle these fast emerging challenges. Executives at one mining company, for instance, created in just a few weeks a global cash-flow tool that integrated and analyzed data from 20 different mines so they could effectively strengthen the company’s solvency during the crisis.

Conversely, in some instances, previously business-critical models have been temporarily benched due to changing priorities during COVID-19 lockdowns. Such was the case at most banks across one European country as they redirected outbound sales representatives from branches and call centers to offer support to their customers (for example, helping customers shift to online banking or secure emergency loans). This temporarily erased the need for a tool that these sales representatives had been using daily to recommend products and identify cross-selling opportunities.

Determining where to focus model assessments requires identifying the new business strategies the pandemic has demanded. This exercise should occur at regular intervals (typically every few weeks) among business and analytics leaders until a relatively more stable next normal establishes itself.

For organizations with hundreds or thousands of models supporting critical business, performing a layered model-triage process can focus limited analytics resources on the most critical models. To identify those models, financial-services organizations, for example, typically consider how a model is used, whether a model is material in financial reporting, to what degree COVID-19 has had an impact (in the next section, we discuss how to determine this), and how direct that impact is on critical business. This has led banks to focus on credit, pricing, valuation, and market-risk models, including high-impact macroeconomic models, over fraud or marketing models. Other considerations include regulatory focus or the potential of upstream models to “contaminate” other critical models.Reevaluating model risks to identify problematic analytics

In general, no analytics system is immune to faulty predictions or suboptimal recommendations resulting from changing patterns of behavior captured in the data feeding into the models. At the same time, not all systems will face the same level of impact, and companies in different industries will find the most problematic models in different pockets of their organizationsReevaluating model risks to identify problematic analytics

Understanding which models to trust and which to scrutinize for problems requires understanding the factors that influence model performance:

• How a model is built. The more complex the model, the more opaque it is, so leaders have more difficulty gauging when its predictive powers may be in question. In cases where the simplest possible model is a complex one, explainability tools can make opaque models more transparent to reduce risk and enable humans to incorporate their expertise and judgment to address highly volatile elements.
• What data are used. Some types of data present more potential problems for the models that rely upon them than others. For example, customer-segmentation and -propensity models can be highly susceptible to performance issues, because they typically rely heavily on individual characteristics and behaviors drawn from years of historical customer data, which might look very different from the data coming in from customers today. Conversely, asset-management models that depend on explicit macro- and microeconomic variables can take into account the impact of COVID-19 by simulating the effect of changes to macroeconomic variables, making them somewhat less at risk.
• How features are weighted. Data scientists often shape more accurate predictions by identifying and ranking “features” in the data. Common examples in the case of underwriting would be time to repay and delinquency rates. Data features can be heavily influenced by current events. Bank leaders, for instance, are finding delinquencies occurring significantly less often than expected, given a high unemployment rate, thanks to the direct and indirect effects of government financial-assistance programs. As a result, models using these artificially reduced delinquency rates may offer falsely optimistic guidance after the government interventions end, and that guidance would affect a broad range of underwriting, customer-service, and capital decisions.
• What assumptions were made during the model build. The energy company mentioned earlier was using analytic models that didn’t account for crude-oil prices reaching ?$30 per barrel. Many banks’ models don’t account for negative interest rates. Retailers’ logistics and demand-forecasting models didn’t account for having to move their entire operations to digital. Logistics companies’ forecasting applications didn’t account for the collapse of passenger air travel, which is closely intertwined with air-freight availability and costs. Indeed, all of these were reasonable assumptions before COVID-19 but are no longer accurate today, and that can have tremendous implications for model predictions.

While analytics professionals should handle the actual evaluation of these elements, senior executives and business-unit heads need to understand what they are and how they affect model performance. Only then will they know what to prioritize, how to address the underlying business assumptions, and what questions to ask analytics teams to ensure that the factors are properly addressed—all of which will give them renewed confidence in the models, once they have been adjusted.

Expanding the organization’s data sources

Whether or not a model needs to be rebuilt or retuned, there’s a good chance that a problematic model needs new, fresher inputs to provide more accurate insights in this volatile period. While these are clearly unprecedented times, we find that much data exist that can help organizations better understand current trends and customer needs. Business leaders charged with sourcing the data—whether a business head, chief data officer, or other executive—should expand their sources in a few key ways:

• Incorporate new or previously unused internal data sources. Examples of such data include web-page navigation and mobile-app usage from new (or enhanced) digital channels, as well as transaction data. The North American bank mentioned earlier, which had been relying on traditional credit scores to assess an individual’s credit risk, began analyzing current-customer account data to uncover in real time any gaps in direct deposits or receipt of unemployment checks.
• Use existing data in new ways. A major automotive-engine manufacturer began leveraging telematics data from its engines to improve its understanding of traffic patterns as cities reopened and to inform demand forecasts. Previously, the telematics data were used chiefly to support maintenance and warranty work.
• Increase the frequency of data collection and processing. Sometimes the challenge is not purely what data you’re collecting and processing, but how often you do so. Simply changing the frequency from, say, weeks to days (or even daily) can shorten learning cycles and enable organizations to pivot more quickly as events on the ground occur. One financial-services company moved from monthly to daily liquidity-reporting processes during the crisis, so leaders could immediately identify any funding gaps.
• Acquire external data. In some cases, the acquisition of external data might include open-source data, such as public-health and location-specific data to, for instance, anticipate workforce availability and potential supply-chain disruptions. In others, it may necessitate buying data previously considered too expensive or difficult to obtain. Banks hesitant in the past to spend the time and money to obtain customer permission to access banking records from other banks through open-banking regulations can likely justify the cost today. Finally, external data may also come from novel—and potentially even surprising—partnerships, as in the case of several original equipment manufacturers that shared data on geographical product-demand trends. By doing so, these companies were able to build a more holistic view of market demand, so each could ensure it had sufficient inventory

Promoting faster cycle times for decision making

The benefits of agile are numerous and well documented, and it’s easy to see how such an approach helps companies today in their scramble to adapt. The faster an organization can detect unexpected market changes, test new ideas, and adjust, the more successfully it can respond. One telecommunications provider uses agile practices to deploy analytics-driven micro-campaigns daily, evaluates the results immediately, and then fine-tunes campaigns the following day. This rapid-fire approach recently helped the company recognize quickly that one of the models informing the campaigns was not accounting for the spike in remote working. By updating its algorithmic models to account for this shift, it better predicted and responded to the need for additional products, including personal Wi-Fi hotspots—a product area for which it began capturing greater market share.

• One critical enabler of this effort was the convening of a multidisciplinary team, a fundamental component of agile delivery practices. We find companies are best served when they build interdisciplinary teams with a diverse set of roles, including domain experts, frontline users, and technical staff, to develop new analytics capabilities for the business. By incorporating these varying perspectives into one team focused on a critical activity and model, leaders can better ensure all aspects of the model’s risks—including data-fit issues, the analytical technique employed, and the functioning of a user dashboard—are addressed.

How to ready your company for whatever comes next

Ultimately, a vaccine for COVID-19 will be developed. The next normal will emerge, and leaders will move from survival mode to more solid ground. This will be a time for reimagining the business and for reform. At that point, how can you structure analytics so your organization doesn’t face the same model challenges coming out of the pandemic that surfaced going into it? Five steps in particular will help organizations respond more flexibly:

1. Deploy a digital nerve center. Digital nerve centers have emerged as a core capability during the COVID-19 response, enabling organizations to mobilize resources, including new data sources and analytics capabilities, to help business teams get a handle on emerging trends quickly, shorten feedback cycles, and gain greater insight into the various scenarios they might face. A retailer with grocery stores in 15 countries is using a digital nerve center to provide critical business functions—supply chain, employee protection, finance, customer and store operations, and digital channel operations—with rapid access to insights about the company, customers, and suppliers during the pandemic. Already, the work has enabled supply-chain leaders to keep store shelves well stocked, even for high-demand items.
2. Embrace real-time data. Previous articles have outlined in depth the shifts necessary for building a data architecture to drive innovation. One shift in particular—the move to real-time collection and analysis of website, social-media, clickstream, and app data—has taken on increased urgency in recent months. Leaders no longer can afford to wait days and weeks for the “latest data” to arrive. Technologies such as messaging platforms and stream-processing capabilities enable the processing and analysis of data in real time, allowing decision makers to respond in hours rather than days or weeks.
3. Prioritize cultural shifts. Many leaders found during the pandemic that their companies could be more agile during a crisis than they had realized. Interdisciplinary teams, agile ways of working, and data-driven mindsets sprouted overnight and produced highly targeted and fruitful analytics capabilities. Cultivating these shifts—through, for example, reskilling workers—will be paramount to keeping the momentum going. Such work is possible even while employees are working from home. One financial-services company has launched Zoom-based training to upskill senior company leaders on basic artificial-intelligence concepts, possible ways to use the technology, and techniques to drive change, preparing the company for the next normal.
4. Ensure compliance by design. Various activities and tools can help teams ensure that critical oversight is baked into the analytics development process, so risks are continually managed and detected and rapidly addressed. These include documented guidance and checklists on topics such as setting up diverse teams, ethics training, risk metrics to use, and changes to keep an eye out for—for example, changes in laws, regulations, and policies, as well as evolving social norms. Activities include putting in place methods and data tools for detecting and mitigating risk in data and monitoring models.
5. Invest in diverse teams. COVID-19 has affected different communities in very different ways. Diverse teams can help organizations predict the changing needs of their customer communities significantly faster and more accurately. (In fact, an abundance of research shows that diversity is linked to higher financial perfroamnce.) We find companies are best served when they build interdisciplinary teams that include gender, ethnic, cultural, and geographical diversity, along with a diverse set of roles and perspectives, to develop new analytics capabilities for the business. From a role-diversity perspective, individuals with translator expertise are increasingly vital during this crisis to bridge the business and analytics realms and effectively link data, models, and insights to the business problem they are meant to solve or the decision they are meant to inform.

While advanced analytics has demonstrated value before and during the COVID-19 pandemic, it has also shown fallibility in the wake of a crisis. Leaders who recognize the sources of fallibility and take steps to strengthen their company’s analytics strategy will be better equipped during the present crisis and future extraordinary events.

Thanks

---------------------------------------------------------------------

Excerpts borrowed from Mckinsey research and articles