The pandemic made 2020 for restaurant delivery. If dine-in recovers, is a rough patch coming?

The COVID-19 pandemic disrupted how consumers eat, driving us away from on-premise dining towards delivery platforms like DoorDash, UberEats, and GrubHub. My colleague Elliot Oblander and I spent much of the past year uncovering how, how much, and why the COVID-19 pandemic transformed customer behavior in the restaurant delivery category, to better understand what consumer demand might look like after the pandemic subsides.

We decided to study this question because it's an important one. The prospects (and for many, survival) of restaurants and restaurant delivery companies hinge on what consumer demand will look like after the pandemic ends. As of December 2020, fully 17% of all restaurants in the US had closed permanently or long-term (link), with another 14% saying they would need to close without significant incremental federal aid. This is a not well understood yet top-of-mind issue for executives and investors (and media -- for example, this interview after DoorDash's Q4 earnings). Given the pandemic is only beginning to be contained, we hope that our analysis can provide some early insight. [Also, while many questions are important, we believe our collection of data sources allows us to answer this one! ]

We shed light on these questions by first inferring what the delivery category would have done if there was no pandemic in the first place. Unlike prior COVID papers, however, we predict not only overall revenue, but also the decomposition of revenue into its customer-driven drivers -- customer acquisition, retention, ordering, and spend (AROS for short). Blue Apron had no issues with revenue growth at the time of their IPO, but as I had shown, a proper AROS analysis implied troubling issues with their unit economics. It pays to go "beneath the surface" to understand how revenue comes about.

We use this both to understand what the category may (partially!) "mean revert" back towards when consumer demand normalizes post-pandemic, and also to provide us with a "pre-pandemic baseline" off of which we can estimate the overall impact of the pandemic. Finally, we move from “what” to “why,” uncovering the different mechanisms driving these impacts -- in particular, how delivery demand was influenced by changes in employment, stay-at-home behavior (e.g., because of working from home, being afraid to go outside, or self-quarantining), store closures, and substitution from on-premise dining (e.g., because of government restrictions or consumer fears of going on-premise because of health risks).

For those interested in the deep dive, you can download the full paper here.

But if you would like a short summary of the main highlights, they are as follows:

1. COVID made the year for restaurant delivery. Industry-wide delivery sales went from ~$23B in 2019 to ~$51B in 2020, implying growth of ~$28B. About $19B of that $28B (70% of total) was purely due to the pandemic. If it weren't for the pandemic, sales growth would have fallen by over half over the prior year.
2. That growth was largely due to people replacing on-premise restaurant visits with delivery orders, either because on-premise dining wasn't allowed due to dine-in restrictions, or because they were afraid to go on-premise for health reasons. The other major mechanisms -- employment changes, store closures, and individuals staying completely at home -- all depressed delivery sales, all else equal. Substitution from dine-in more than compensated for those sources of weakness.
3. Logically, then, if dine-in activity were to return to pre-pandemic levels, our analysis would suggest a possibly significant headwind for industry-wide delivery sales. That's just how substitution works. If delivery growth was due to delivery orders replacing on-premise restaurant visits, then if on-premise restaurant visits come back, those purchase events will not be occurring elsewhere.

Next, I'll describe what we did in a bit more detail, unpacking our data, our approach, how well our model validates, our results, and additional discussion of the implications of the work.

Our data

We were able to carry out our analysis by obtaining great data from great data companies.

Purchase and location data: First and foremost was credit/debit card data from Earnest Research, one of the biggest providers of this and other so-called "alternative data". Through Earnest, we observe credit and debit transaction activity, possibly across multiple cards, for 1.8 million panel members in the restaurant delivery category from the beginning of 2016 through the end of 2020. Importantly, we also were able to obtain the modal location of these panel members every month. This allowed us to observe spending in the delivery category, as well as how it varied over time and across geographies.

This data was very informative -- while they do not observe spending for every single individual in the US, raising the specter of non-representativeness issues, we take a page from our prior paper's playbook (now in Marketing Science) and compare delivery sales as observed through the panel data with "gold standard" data directly disclosed by DoorDash and GrubHub through their SEC filings. A simple log-log regression between panel and SEC data sales had exceptionally strong goodness of fit. For example, there was a near-perfect R^2 of 99.8% between panel and SEC sales for the sum of DoorDash and GrubHub sales, giving us confidence that the category-level trends observed in the panel can be very easily translated into what is happening at the population level:

Market entry: Remember that the first thing we predict is customer acquisition. People become prospects when they're able to adopt, and then some time elapses until they do adopt. As an important first step then, we need to know when people become prospects. All prior literature (including my own) has assumed that everyone in a given country is a prospect at the time the company being analyzed begins commercial operations, but that is a particularly bad assumption for a business as local/regional (and with such a staggered rollout) as restaurant delivery. In our case, individuals become prospects when at least one delivery company operates in their area.

For this reason, we pulled together two data sources that, together with the aforementioned Earnest data, allowed us to understand when delivery companies began operating in different parts of the US -- the Wayback Machine and YipitData.

The Wayback Machine archives historical versions of companies' websites. Well, as it turns out, from 2004 to 2015, many major restaurant delivery companies (including GrubHub, DoorDash, and UberEats) reported on their websites every single city they operated in. For example, this is from GrubHub's website on October 7th 2011:

We systematically went back through all cities each major delivery company operated in to find the very first time that each city was entered, which we used as the market entry date.

YipitData provided us with more recent data. From October 2018 through December 2020, they provided us with the name and location of all restaurants listed on all major food delivery platforms, allowing us to directly observe which markets were entered at which times by which companies. Consider GrubHub in Bakersfield, Lubbock, and Springfield, for example:

Analyzing this data across all major platforms over time makes it easy to see which cities were entered more recently. [We could also directly observe the total unique count of restaurant listings by platform, which we'll touch on in the "Wonky Comments" section below.]

This leaves the pesky issue of cities which weren't present in the Wayback data (i.e., were entered into after 2015), but were present at the very beginning of the Yipit data (i.e., were entered into at or before October 2018). For these markets, we went back to the Earnest data to infer market entry (because we have purchase location data, albeit for a subsample of consumers, through this data source).

Restaurant supply, dine-in activity, and stay-at-home behavior: Last but not least, we needed proxies for the available supply of "restaurant", how much people are dining in, and how much people are staying at home, over time and across geographies. This would allow us to understand how these behaviors influence spending behavior in the restaurant delivery category.

To this end, we use data from SafeGraph, a leading geolocation data provider that observes cell phone data for 18 million devices. They directly provide stay-at-home compliance data by census block group-day. They also provide the duration of time that individuals spent at 945 thousand restaurants over time, from which we could infer how many people dined in and how many people were employed at different restaurants over time.

This is a table summarizing the various data sources:

Our approach

A complicating factor, as we know, is that the pandemic on some level affected everything at about the same time. As such, our approach is to take, well, two approaches:

1. We use an event study approach to estimate the overall national impact of the pandemic on the delivery category. We took all the data before the pandemic started and ran it through a highly validated series of AROS statistical models (similar in spirit to what we used in my prior Journal of Marketing Research paper with Peter Fader) to predict what would happen during the pandemic -- this formed our counterfactual pre-pandemic baseline. After validating the underlying statistical models, we then compared the resulting predictions to what we actually observed during the pandemic, making the gap between the two (a noisy realization of) the estimated effect of the pandemic.
2. We then use a fixed effects regression model to understand the mechanisms driving changes in delivery category spending (and how it breaks down into AROS). This allows us to understand the impact of changes on the margin in the aforementioned drivers -- unemployment, stay-at-home behavior, store closures, and substitution from dine-in activity.

As we discuss at length in the paper (Section 3), these two approaches are very complementary to one another. The former gives us insight into pre-pandemic trends and the overall impact of COVID-19 (albeit without much insight into mechanisms driving those changes), while the latter gives us insight into mechanisms (albeit without providing us with valid insight into the overall impact).

Absolute impact analysis

It can be helpful to first see the data as it sits before running any models. This chart shows total customer acquisitions, total active customers by cohort, average orders per active customer, and average order size for the delivery category as observed through the Earnest data:

We can see COVID affected different customer behaviors very differently:

1. After peaking in March 2019, customer acquisitions had been steadily falling for 9 months. COVID temporarily drove a lot of people to adopt, but acquisitions are now falling once again. The category is fairly saturated -- 50% and 62% of all the panel members adopted by the end of 2019 and 2020, respectively.
2. Using the count of customers within a particular cohort as a proxy for retention, it doesn't appear that retention has changed much.
3. Orders per active customer and AOV both had been very well-behaved pre-COVID, but spiked in sustained ways during COVID.

It is these behaviors, roughly speaking, that we look to capture through our AROS models. But given how well-behaved these trends were pre-COVID, and how well-validated our AROS models are across a variety of different companies (more on this in my Practice Prize video!), we know that our models should validate well.

And they do. [ For more on validation, see Sections 5 and 6 of the Web Appendix.]

Results. The following figures show the impact of the pandemic on acquisitions, orders, AOV, and sales (light blue = actual, darker dotted blue = pre-pandemic baseline expected + confidence interval):

Given the model-free evidence we just showed, these charts should come as no surprise.

The gap between the two lines represents the impact of the pandemic within the panel. We then use the aforementioned log-log regression to convert this into the impact of the pandemic on the overall US population (i.e., at the national level). The figure below summarizes these national-level effects:

Breaking it down with simple math, we estimate total US delivery sales of ~$23B in 2019. Delivery sales jumped to ~$51B in 2020, an increase of ~$28B. Our results suggest fully ~$19B of the $28B (70% of the growth) were caused by the pandemic (i.e., those sales would not have occurred had there been no pandemic). My best estimate is that year-on-year sales growth for the delivery category would have fallen by about 50% if it weren't for the pandemic.

The rest of the tree diagram above shows how this sales came about. In short, what the pandemic did is get people who adopted into the category before the pandemic began to order more frequently (people were stuck at home, and if you remember, were made to feel almost patriotic for ordering delivery) and spend more when they order (think family orders). While there was a jump in the number of people who tried delivery for the first time, abnormally high customer adoptions had a limited role to play in the delivery "COVID bump."

Analysis of mechanisms

We wanted to also better understand what was driving these dramatic changes in behavior -- what was the effect of shifts in employment? People staying completely at home? Restaurant closures? Wanting to go to a restaurant but being unwilling or unable to?

To better understand this, we ran a fixed effects regression model, leveraging the fact that different parts of the US were affected by each of these factors differently, both on the way down to the COVID trough in March, but also (even more so, as it turns out) in the subsequent rebound. Some cities were subject to more lax dine-in restrictions than others. Some cities were more exposed to COVID-related economic weakness (e.g., areas where travel was a big driver of the local economy, versus tech hubs). And so on.

Case in point: Jacksonville Florida, New Haven Connecticut and Honolulu Hawaii:

[ What data are we reporting in this figure? This is a summary of the regressors: ]

Honolulu had a severe unemployment spike while New Haven's was delayed (top-left). Jacksonville and New Haven had similar drops in dine-in, but New Haven double-dipped (bottom-left). And so on. It's these differences relative to the national average that identify the model. We include day and city-specific fixed effects to control for national trends, so the variation that we exploit is within-city, within-day. On the margin, if a certain area saw an uptick in dine-in relative to the national average, for example, the partial effect of that on delivery sales (controlling for the other regressors) is what drives our coefficient estimate for dine-in.

For those interested, we discuss the identification of this model and related endogeneity concerns such as simultaneity of dine-in activity with delivery sales, of dine-in activity with stay-at-home behavior, and of restaurant employment with delivery sales, as well as a number of confounds/alternative mechanisms, including socioeconomic status, political beliefs, population density, stay-at-home restrictions, strategic targeted marketing by delivery companies, and the number of restaurant listings on delivery platforms is available in Section 5.2 of the paper and Section 9 of the web appendix. We also include a number of robustness checks in the web appendix.

Results. We separately ran the aforementioned fixed effects regression for four AROS variables for the delivery category: acquisitions, total orders, AOV, and how it all rolls up into sales. This table summarized the regression estimates:

In the interest of space, I'll just focus on the results for total delivery sales. Basically, the increase in delivery sales during COVID was due to substitution from on-premise dining. In layman's terms, when people were unable to or were afraid to dine in, on the margin, delivery sales went up and vice versa. Unemployment, people staying at home more, and store closures all held down delivery sales during COVID.

Implications. So what does all of this mean? A few things...

For one, this analysis goes to show just how powerful an effect substitution away from on-premise dining was during the pandemic. Swapping out on-premise dining and replacing it with delivery orders is what drove the delivery boom. And then some -- the other mechanisms, including falling income and staying at home, all served to depress category-level sales on the margin, so this substitution effect more than compensated for all of these other factors.

If true, then it only stands to reason that if on-premise dining activity were to return to pre-pandemic levels, the delivery category will face significant headwinds. Substitution cuts both ways, and there are only a few ways that you can put dinner on your table each night, with delivery and dine-in being two of them.

It also implies that COVID was basically a big gift for the delivery category that the restaurant industry was effectively forced to pay. This doesn't mean that the delivery companies did anything nefarious -- they even gave a portion of their windfall back to the restaurants (as did regulators through commission caps), but it is what it is.

For two, what a boom it was. It would have been one thing if COVID was responsible for 20% of overall sales growth in 2020, but 70%? That's a lot! And so, to the extent that there is an even partial normalization to pre-pandemic trends, sales could fall, at least temporarily, by a lot.

Caveats

While there is a whole section of our paper listing out various caveats, I also wanted to highlight a few important ones, especially given what I typically write about here.

First, the resumption of dine-in will be a process. It seems unlikely that dine-in will come back with the flick of a switch. Regulations restricting dine-in will be an important first step, but we won't see a meaningful move back until the vaccine rollout is more complete, the dust has settled, and importantly, when the restaurants themselves have normalized their own operations in a number of important ways. In addition to laying off waitstaff, many restaurants changed the configuration of their restaurants to lean more heavily into delivery. As diners come back through their doors, some of those changes will impede the on-premise customer experience. For example, waitstaff will need to be re-hired, which has been perhaps surprisingly difficult (there is an interesting chicken-and-egg situation here, especially for waitstaff that traditionally rely heavily on tips that won't really be coming until dine-in comes back). Restaurants will be happy to make the trade back to much higher margin in-store business, but at some point, for many, it will require emphasizing delivery less. These things may take time. We are not saying it will be immediate.

Second, this is not a so-called "customer-based corporate valuation" (CBCV) analysis. Those of you who follow my work know that I have done some preliminary CBCV-inspired work on DoorDash, and was very open/forthcoming about their constructive unit economics (tldr: they don't spend too much to acquire customers, and while variable profit per order is very small, both profits and orders grow a lot as customers age before they churn). I stand behind those inferences and these results do not contradict those. Pre-pandemic unit economics were good at DoorDash, and the pandemic only helped. Also, the repeat purchase trends evident at DoorDash and for the category as a whole were very consistent with one another -- older cohorts are consistently buying more over time in general, after an initial shakeout in the first couple of months. DoorDash had good unit economics because of good industry-wide trends, and because DoorDash is significantly outperforming the rest of the industry, as they noted in their SEC filing summarizing Q4 results (in particular, this chart). In this analysis, we are not looking to infer industry-wide unit economics, or model idiosyncratic individual-company performance (or competition for that matter). Our goal here is to understand pre-pandemic industry-wide consumer demand trends, and how and why the pandemic influenced them.

All that being said, we're also not shying away from the implications that these results may have for where delivery sales may be heading, especially in the medium term. If the impact of COVID is both very large and primarily driven by substituting away from dine-in, then if dine-in comes back, our analysis would suggest a big headwind is coming for delivery sales.

Wonky comments

Listing behavior. You might be wondering - how did restaurant listing behavior change as a result of the pandemic? We had full visibility into this through Yipit's data, which as mentioned earlier, provided us with every single restaurant (name, street address, etc) listed on every single major platform, every month from October 2018 onwards. Through this, we could get the total number of restaurant listings each month for each platform as well as for the category as a whole (after deduplication).

Here is the number of restaurants for each individual platform:

And here is how the results roll up to the category as a whole:

Interestingly, this implies that the pandemic didn't really cause a spike in restaurant listings, which, if it did, could conceivably contribute to the surge in delivery demand that we saw. But in fact, if anything, there was a decline in listings (for GrubHub and DoorDash -- Uber Eats and Postmates continued along their pre-pandemic trend).

Those declines are not a data error -- they are real, and stem from the fact that GrubHub and DoorDash listed many restaurants without explicit consent from restaurants, something they have gotten a lot of flak for. When the pandemic began, many of these so-called "non-partnered restaurants" closed for extended periods of time, making it unclear to the platforms whether or not they were open. To lower the risk of negative customer experiences stemming from customers placing orders at restaurants that were closed, the platforms elected to temporarily remove some non-partnered restaurant listings, driving a temporary decline in listings that reversed itself in subsequent months.

In any case, this rules out restaurant listings as being a driver of delivery sales during the pandemic. It also could be an indication of supply-side saturation. One would have thought that the pandemic would have driven many restaurants who had held off on delivery to sign up for the first time. The fact that listings basically continued along their pre-pandemic trajectory implies that whatever abnormal bump there may have been was completely offset by restaurant closures.

Marketing spend. One could also argue that perhaps some of increase in delivery sales was because the delivery companies decided to open the pocketbook and ramp up spending on advertisements and other marketing activities. But from the limited view that we have into marketing spend, this does not appear to be the case. The year-on-year change in GrubHub and DoorDash's quarterly marketing spends (as reported in their SEC filings) was no different in Q2 2020 than in prior quarters. In retrospect, this makes intuitive sense - consumers were beating down their doors and the bulk of the boost in pandemic delivery sales was from existing already active customers, so the delivery companies had little need to spend on marketing to generate those sales.