How Machine Learning in Business is Affected by the COVID-19 Pandemic

About

David Mosen is Chief Data Scientist of Crayon's Data & AI Center of Excellence. He has carried out cancer and epidemiology research at Harvard University and Imperial College London. He also has wide industry experience, having offered AI consulting services across the US and Europe. As a technical lead, his current focus is on end-to-end deliveries of AI-powered cloud-based solutions, implementing methodology and building reusability across applications, markets and verticals.

Introduction

COVID-19 has disrupted every single social, economic and political aspect of life and it’s difficult to predict how the crisis derived from the pandemic will evolve. Yet, it is clear that some of the effects will stay. Shifts in mentality will be forced upon many organizations, especially on aspects like policies on remote work and the need for travel. Some, mostly smaller, businesses will not resist the financial stress on their accounts and will either default or be pushed into mergers and acquisitions. It is even likely that we witness the appearance of new business models, potentially more efficient in their activities, driven by the necessity to adapt to limiting circumstances.

As abstract a concept we might think it is, machine learning is directly connected and influenced by the real world, so it is decidedly not disconnected from such circumstances. This is especially true for businesses that have machine learning models in production to gain intelligence for sales forecasting, stock optimization and customer behavior, among others. Thus, businesses, particularly those that are data-driven and use machine learning solutions, need to be prepared to change.

Machine learning is directly connected and influenced by the real world, so it is decidedly affected by COVID-19. Businesses that are data-driven and use machine-learning solutions need to be prepared.

Dealing with the new reality

The silver lining is that in order for businesses, specifically machine learning practices, to properly respond to the new situation, we can look to the public health outbreak response models as a reference. In the early stages, epidemics like COVID-19’s typically progress almost exponentially. It is usually at some point during this time that the outbreak gets detected by public health authorities, who perform surveillance and qualitative risk assessment. Then, an early response involves wider case detection and intervention planning, followed by its implementation. These responses usually follow a step-by-step process, taking measures based on when tolerable thresholds are reached. Once the outbreak is over, it is time for capacity building, lessons learned and policy formation.

We can map such process to data science and machine learning. Surveillance and qualitative risk assessment translate to a) ad-hoc exploratory data analyses (EDA) and b) evaluation of how the machine learning models will get affected by the changing characteristics and distributions of input data. Based on such observations and conclusions, plans should be implemented to adapt the workflows and models accordingly. Finally, in our context, we can understand capacity building not as human resource capacity, but rather as making machine learning models more robust and tolerant to anomalies in the data that are derived from unique events like an epidemic. From the lessons learned, a set of actions should arise and be adapted into implementation guidelines and preparedness procedures. These include a report of how relevant models are affected, prevention measures (e.g., how to increase robustness to similar events in current models and those in the future) and early response protocols to minimize the effects, stating responsibilities and how each process, data and/or model element of the existing pipelines should be adapted.

To make these concepts more concrete, I will explore how an exceptional event like the ongoing pandemic affect machine learning models currently in production and how it will affect future models that will need to feed on data and events generated during this time. Specifically, I will go through the effects on some common data science and machine learning practices, as well as on specific machine learning applications.

Effects on Common Data Science Practices

A/B testing

A/B test duration should span enough time to enable the modeling of the natural variability of a market cycle. On one extreme of the spectrum, there’s parts of the economy with multi-year life cycles, like real estate investment, where the impact of relatively short-lived events on A/B testing is less relevant. On the other extreme, stock trading, with up to several complete cycles per day, has a solid statistical base in quantitative finance, a field that is accustomed to highly volatile data and to unique events, and thus going beyond simple A/B testing models. In the middle, most e-commerce businesses usually present business cycle periods of around two or more weeks. It’s in this timeframe that samples of A/B tests are likely to present what is known as length pollution, where an external factor, like a pandemic, is continually changing the context. This invalidates the results due to the analysis of inconsistent and unrepresentative behavior. Therefore, the trade-off between sufficient data and the risk of pollution is to be re-evaluated. Also, conclusions will be rendered invalid after a short time, so the benefits and costs of A/B testing during these times need to be evaluated and balanced.

Anomaly detection

Anomaly detection is applied in many domains, some of which are not directly affected by socioeconomic disruptions, like predictive maintenance in manufacturing or medical image analysis. Conversely, other applications need to adapt to such events and navigate the struggle of defining “normal” in times of COVID-19:

• In IT infrastructure monitoring, like e-commerce, there may be sudden surges combined with upward trends. Depending on the need to monitor such events and whether off-the-shelf solutions are used, this should prompt a change in the sensitivity of the time-series-based algorithms and/or alert threshold parameters.
• Now that measures like lockdowns and confinements are in many countries around the world, there is a shift towards online payments and micropayments, altering patterns and overall figures. In the context of fraud detection in finance, this might pose an issue for some applications of transaction monitoring and is an example of what is called “concept drift” in machine learning.
• While employers, customers and business partners are exchanging guidelines and information on the outbreak, spammers are using COVID-19 as bait in their attacks. This happens on top of the fact that for years pharmaceutical spam accounts for a large proportion of the overall spam. Even though this specific application of anomaly detection is better prepared than others, as detectors are used to the mention of current events for email spam attacks, it still gets affected and models need to be kept up-to-date more than ever.

Missing data

A lot of businesses are being forced into shutdowns. In the case of manufacturing and industrial production, one consequence of a halt in production results in missing data for the corresponding shutdown dates. From this perspective, the result can be a single long gap in the collection and storage of data. On the other hand, circumstances like workforce capacity reduction, inconsistent supply chain and reduction in demand force businesses into intermittent production breaks or cutdowns. Because data incompleteness is not a main concern under such state of affairs, future training of machine learning models will have to deal with it. If capacity allows though, to alleviate issues in the future, a log of events and how they are represented in the data during this time should be kept.

Effects on Specific Machine Learning Applications

Demand forecasting and inventory optimization

Situation. The pandemic is affecting business sales in different ways, depending on factors like their industry, market(s) and supply chain robustness. Some organizations are seeing an uptick in their activity (e.g., communication, home entertainment, online education, medical supplies), others are seeing a reduction (e.g., international trade, hospitality, B2B payments) and for many it will become unstable (e.g., essential goods, manufacturing). In addition, the pace at which these changes occur varies greatly and are linked to the evolution of the pandemic.

Problem. Forecasting such events is challenging because of their nature and the limited amount of historical information from which a reference base can be built [6]. Namely, statistical forecasting, from ETS and ARIMA to LSTM-based, has the underlying assumption that historical data provides all the required components to predict ahead (e.g. trend, seasonality, autoregression) and to do so with an acceptable level of uncertainty. However, when it comes to demand, such assumption is violated as soon as the epidemic starts affecting the market. It is precisely during these times that aided judgement and interactive forecasting is most helpful. An interesting instance is “forecast by analogy”, where a model is used to analyze the behavior during the current outbreak based on previous outbreaks. This could be synergistic with the already deployed machine learning models, putting exceptional care on defining uncertainty levels. On top of this, rapid-reporting cycles will show how the business is affected, and what changes should be made such as in inventory optimization to stabilize the supply chain.

Problem. From a historical data perspective, if the issues derived from the outbreak are short-lived, the best approach to deal with data irregularities might be to omit or even impute them, as alteration to the original time series would be limited. This is especially true if forecasts are quite granular (e.g. at a daily level), relevant data is amassed quickly and trends present a high frequency. However, it is likely that the ripples of the outbreak will extend well beyond a few months into the future, while many forecasts affected by the outbreak need to model trends and seasonalities that encompass weeks, months or even longer. Depending on the scenario, using shorter training history or domain-specific a posteriori corrections might be more appropriate. The latter could take the form of manual adjustments or, ideally, normalizations that leverage domain-specific knowledge.

Product recommendations in e-commerce

Situation. Overall customer behavior in e-commerce is drastically changing. This is not limited to stores with sanitizing and contagion prevention items, or even to food and essential household products, as there is a wave of indirectly affected providers of all goods and services. The developing buying patterns cannot be assumed to be linear either, as each phase of the epidemic might come with its own patterns. The sales records can also be expected to be highly volatile, depending on factors like changing household budgets.

Problem. Machine learning models that recommend products based on data prior to the reaction to the epidemic will rapidly become obsolete, while most such models assume stationarity. There are two non-mutually-exclusive paths that will lead to such problems: Model staleness and relatively long historical records used for training. The frequency with which models are re-trained needs to be re-evaluated, while shorter training should be considered, even if under general circumstances this results in poorer performance. The bottom line is that models and the pipelines that build them are in better shape to face this problem if they are designed to be agile and evolvable. For those that are not as flexible, shorter-term adaptations are prescribed, bearing in mind lessons learned.

Takeaways

Though we are in an uncertain time, I have three main points to reiterate when it comes to machine learning in business during this pandemic:

Planning. Where relevant, teams should plan ahead and perform revisory EDA to figure out how to adapt ML models, making them more robust to ongoing events. They should also take the chance to develop guidelines to improve preparedness and minimize the effects of unique events on susceptible models.

Agility. If technically feasible, statistically accommodatable and economically sensible, models in multiple business applications should be retrained more often on less data. This is most notably important for where there is a strong temporal component. The purpose is to become more agile and better capture the quickly-changing nature of the real-world socioeconomic dimension they model.

Retrospective. Historical data is affected. Missing values, potentially intermittent, are introduced. So are altered patterns not representative of future trends and behaviors. Reasons, associations and expected consequences of such data issues should be documented and, where possible, minimized, so that they are dealt with in the future in the best way possible, be it by omission or by correction.