Measuring the Accuracy of COVID-19 Prediction Models (Update #6), and the State of the COVID-19 Pandemic

Justin Gabriel and Alkiviadis Vazacopoulos, Stevens Institute of Technology 

           In the past week, COVID-19 cases in the United States have continued to increase, driven in part by a surge in cases from states that have attempted reopening in the last month, such as Florida. As the United States continues to lag behind many European countries in terms of controlling the spread of COVID-19, we continue to emphasize the need for greater testing and paying attention to forecasting models, which can help policymakers make decisions and better prepare us for future pandemics. In this update, we continue to discuss the accuracy of various COVID-19 projection models for the last week, as we have done in previous updates, testing rates across the United States, and Wunderman Thompson’s Predictive Recovery Index.

Updates for Reich Lab Models

           Below we show the mean absolute percent errors (MAPEs) of various select COVID-19 prediction models for July 12, 2020 (MMWR Week 29 of the pandemic), one week in advance, in predicting total COVID-19 US deaths. Johns Hopkins University reported about 135,000 cumulative COVID-19 deaths as of July 12. As with previous updates, information about these models comes from the Reich Lab Forecast Hub.

This week, the most accurate model was the DELPHI model created by the MIT Operations Research Center, with a MAPE of 0.156%. Just as we detailed in our Update #4, where the DELPHI model was also the most accurate, the DELPHI model is an SEIR model that also attempts to account for under-testing and government intervention. The second most accurate model was the University of Arizona’s EpiGro COVID-19 Model, with a MAPE of 0.348%, which is based on an earlier disease forecasting model, EpiGro, with the parameters, changed to match COVID-19. The third best model was the MechBayes model of the University of Massachusetts-Amherst Influenza Forecasting Center of Excellence & College of Information and Computer Sciences, with a MAPE of 0.374%.

Policy-Based Models

           As stated in our Update #4, we have also switched to using the policy-based models maintained by the Institute for Health Metrics and Evaluation (IHME) rather than the MIT-run models we reported on previously. Below we show the MAPEs of the IHME model in predicting death counts in six different states and five different European countries for three different scenarios a week ahead, just as we have previously done with the MIT models. For the IHME models, in Scenario 1, masks are used at a rate of 95% in all public locations, and “Mandates are re-imposed for 6 weeks if daily deaths reach 8 per million.” In Scenario 2, “Mandates are re-imposed for 6 weeks whenever daily deaths reach 8 per million.” In Scenario 3, social distancing measures are continuously eased without future restrictions. These are indicated on the IHME website as “Masks,” “Projection,” and “Easing,” respectively.

        The fact that the same MAPEs were calculated for each scenario in each state/country indicates that at least one week ahead of time, the IHME model does not see much difference between the various scenarios. However, while short-term the differences in death counts in the short term between a scenario in which everyone wears masks vs. a scenario in which most people don’t wear masks may be minimal, the model clearly demonstrates that in the long-term, these policy differences do matter. For example, below we show an IHME projection for death counts in Texas from July 12, 2020, onwards. Note that while in the short-term, which our predictions are focused on, there is very little difference between the policies, there is a large difference between projected death counts in fall months such as October or November depending on the extent to which masks are worn by the government. This underscores the need for everyone to wear masks in public, as it is a simple yet effective measure that can reduce the number of COVID-19 deaths in the future.

We also note that the IHME models actually underestimated the number of deaths that would have occurred in New York by this week.

Testing Rates Across the Country

           Below we graph daily COVID-19 positivity rates for the United States as a whole and in six states: California, Florida, Georgia, Michigan, New York, and Texas, over the last two weeks. We note that the World Health Organization recommends that positivity rates remain below 5% for at least two weeks in order for an area to declare that an epidemic has been controlled.

           Only Michigan and New York among the regions sampled have maintained their daily positivity rates below 5% over the last two weeks, indicating flattened or nearly flattened case rate curves. The United States national average continues to hover around 8% with no signs of decreasing in recent weeks. Positivity rates in Texas are extremely volatile but are consistently above the national average, and Georgia and Florida have among the highest positivity rates in the country, with some experts calling Florida one of the global COVID-19 epicenters. To further demonstrate this point, we compared daily positivity rates over the last two weeks for the two largest counties in Florida (Miami-Dade and Broward) with the two largest counties in New York (Kings and Queens).

           Kings and Queens counties, both of which are situated in New York City, a former global center of COVID-19, have virtually flattened their case rates, with both counties having about 1% positivity rates over the last two weeks. This is not to say that New York policymakers should get complacent or over-confident: there are still concerns of multiple waves of COVID-19 affecting any area in the future as social distancing restrictions ease, and there may still be under-tested areas. But the New York counties stand in stark contrast with the Florida counties, with Broward having positivity rates consistently around 20% and Miami-Dade consistently over 20%. There are no signs that COVID-19 is slowing down in Florida and across the country’s Sun Belt, emphasizing the need not just for more testing, but also for stronger social distancing measures and the rollback of reopenings until adequate testing can be provided to get daily positivity rates at a controllable rate.

Wunderman Thompson Predictive Recovery Index

           Finally, we report on Wunderman Thompson’s Predictive Recovery Index, a model developed by marketing agency Wunderman Thompson in collaboration with IBM. Wunderman Thompson and IBM have developed three indices that are designed to assist businesses in reopening and monitoring the state of the pandemic on a county level. These indices are the Risk Index, which is based on CDC-defined risk factors such as cancer, age, and diabetes rates in a county, the Readiness Index, which depends upon hospitalization rates, and the Recovery Index, a machine-learning-based index that is calculated by comparing consumer spending at a given point of time with the spending at a similar point last year. The Recovery Index is designed to predict and model recovery from the economic fallout of COVID-19 rather than medical recovery. The calculation for Risk Index means that it does not change much in the short-term and that the indices are calculated differently and are therefore not comparable with one another. We also found after tracking the model daily for New York and Florida that county-level calculations for Readiness Index do not change much daily either.

           For example, below we show the Recovery Indices for New York and Florida as calculated by Wunderman Thompson on July 14, 2020.

        This is a tool that can be useful for businesses, governments, and other institutions in making decisions regarding COVID-19, as the website also includes useful data used to calculate these values, such as the presence of COVID-19 risk factors in a county, unemployment rates, and tools which allow for comparisons between counties on any of the three indices.