Promoting pluralism in climate forecasting with prediction markets

Approaches to climate modelling and forecasting should be more diverse but the current emphasis on high resolution GCMs is a product of the way that climate science is funded. If we want more diversity in climate modelling, we must have more diversity in the ways that climate science is funded. When the primary deliverable of research is a forecast, “prediction markets” are an alternative way of incentivizing and aggregating disparate approaches.

A new paper by Marina Baldissera Pacchetti , Julie Jebeile, and Erica Thompson makes the case for pluralism in climate modelling methods — in contrast to the current emphasis on GCMs of ever-increasing resolution. They present machine learning and earth models of intermediate complexity as examples of other approaches that may contribute value beyond what GCMs provide. They also argue for a more “equitable” distribution of research funding to diverse approaches, although they (wisely) leave the discussion of what that might mean for another day.

Recently there has been growing interest in alternative ways of funding scientific research. An article in the Economist on the topic opened with a thought experiment by Michael Nielsen and Kanjun Qiu imagining how science might be done on an alien planet and suggested, “It would be remarkable if the little green men had invented universities, funding committees, a tenure system and all the other accoutrements of modern academic life.” But what might alternatives to the existing system look like? If we assume form should follow function, we should start by asking what is the function of climate science?

The function of climate science is better defined than many other sciences: While most sciences use predictions to test hypotheses, which are essentially their “deliverables”, for climate science the prediction itself is often the deliverable; and when the primary deliverable of research is a prediction, the allocation of funding can be directly linked to its accuracy.

How forecaster compensation should relate to accuracy, especially for forecasts presented in terms of probabilities, has been considered by people like the mathematician I.J. Good. He suggested using the logarithmic scoring rule to reward forecasters. The logarithmic rule is an example of a proper scoring rule which means forecasters should report their genuine beliefs about the probabilities of different outcomes if they want to maximize their expected score (and reward). Forecasting tournaments are often scored using proper scoring rules (e.g. the logarithmic score or quadratic Brier score) although the tournament structure, in which people compete for rank, undermines the propriety of the score: The forecast that maximizes your expected rank might not be the forecast that reflects your true beliefs. So proper scores should not be overlaid with tournaments or targets which distort the incentives provided by the score.

A collection of competing predictions is of limited use to a decision maker.

Furthermore, an equitable system for allocating funding for forecasts shouldn’t reward people merely for providing good information but instead reward people who provide good information that others aren’t providing; this incentivizes diversity. Also, a collection of competing predictions is of limited use to a decision maker who wants a single forecast that, as far as possible, incorporates all the information in the individual forecasts.

A mechanism that both aggregates disparate information and rewards forecasters for making incremental improvements to the collective forecast with novel information or insights is a prediction market. ?While the purpose of most markets is to facilitate the transfer of assets or risks, markets also do something that economists call “information discovery”; they effectively combine information from all the participants and summarize it in market prices. Prediction markets take this information synthesizing function and decouple it from the role of markets in transferring ownership or allocating capital. The primary, and possibly only, purpose of a prediction market is to elicit and aggregate information. ?

Prediction markets have some similarities with recreational gambling, and indeed some prediction market platforms are really online betting exchanges. An important distinction can be made, however: In recreational gambling the rewards to informed bettors necessarily come from the losses of uninformed (or perhaps unlucky) bettors. This arrangement can work for topics with broad appeal, such as sports and entertainment, but for specialized predictions there are both practical and ethical concerns with relying on uninformed participants to subsidize information discovery. Attempts to create betting markets for technical topics, including climate-related ones, have often suffered from low “liquidity” because they are unable to attract uninformed participants in sufficient numbers to make it worthwhile for informed participants to take part.

Subsidized prediction markets are fundamentally different from traditional gambling.

This problem can be circumvented if a sponsor, who seeks information, provides a subsidy instead. This subsidy can be introduced into the market through an automated market maker (AMM). AMMs have been developed which reward participants for the incremental contributions they make to accuracy using proper scoring rules. Expecting participants to be a source of information rather than a source of revenue makes subsidized prediction markets fundamentally different from traditional gambling. It also means that participants need no longer be required to pay to take part, and this means the markets are no longer viewed as gambling by regulators. Laws regulating or prohibiting gambling have been an obstacle to the establishment of prediction markets in many jurisdictions. Instead, these markets can be a mechanism for funding climate forecasting research in a more equitable way.

Over the past six years we have run dozens of pilot prediction markets to test whether they are a viable mechanism for generating predictions of climate-related risks which combine the models and judgements of many expert participants. These markets have typically been far more sophisticated than the simple binary event markets that are common on existing platforms. Some have generated highly granular joint-probability distributions of climate-related variables. Such joint markets could be used to produce simultaneous predictions of greenhouse gas concentrations and climate variables, as well as an implied relationship between them. These markets were not pay-to-play and were instead subsidized by market sponsors including Winton , Lloyds Lab, and Agrimetrics . They have shown that climate experts are able and willing to engage with prediction markets and that these markets can synthesize their evolving views as new information becomes available.

...a new species of entity within the ecosystem of science and academia.

These pilot markets have led to CRUCIAL — an initiative by Lancaster and Exeter Universities — to establish a new species of entity within the ecosystem of science and academia: A platform to host prediction markets for climate risks with expert participants and sponsored by organizations that want funding for climate forecasting to be more diverse and performance-driven.

The longest prediction horizon of any of our pilot markets was just over a year but the aspiration is to extend these horizons to multi-year and eventually multi-decadal forecasts —far beyond the horizons of previous prediction markets. Running prediction markets on such timescales poses new challenges, such as the longevity and governance of the entity operating the markets and how interest is incorporated into the compensation of participants. However, the existence of pensions, mortgages, and 100-year corporate bonds tell us that these problems are not insurmountable. A bigger challenge might be to persuade the funders of climate science, and the institutions that do well under current funding mechanisms, that those mechanisms are not necessarily the most efficient or equitable.

?