What We Learnt from PunterGPT

We built PunterGPT as a kind of light hearted supplement to the more robust and complete statistical analysis built by Macquarie Quant team to pick the winners of Melbourne Cup. Using sophisticated AI language models, we fed in a set of punter conversations from around the track, and asked the AI to analyse it for its best picks based on the often contextual information of the experts that follow horses for a living. The AI picked its top horses based on very different reasons, and one of them ended up winning the race.

Here is our story.

The Beginning

It was only two weeks ago when John Conomos, Head of Global Quantitative Research and I found ourselves talking about the upcoming Melbourne cup. Like all good conversations, it started with banter. Thinking about all the quantifiable details of a horse race, and how the Quant team had turned their attention to forecasting this race using a factor-type analysis. This quantitative modelling approach identifies the factors that are significant for success across horses, and takes a very broad approach for potential drivers. As with all quantitative models, the relevance and significance of factors is identified across horses. In order for a factor to be significant, it must be a useful driver of separation across horses. The assumption is that there are a set of predictive factors, F, that work uniformly across horses, and the role of the statistical analysis is to identify these.

We turned to the idea of that so much of horse races, like financial markets, work on a revolving door of news, context and bits and pieces of information. Much of it is noise, its transient and irrelevant. But some of it is relevant, and specifically relevant to some horses but not others. For example, it doesn't matter if a horse has had international travel recently before the race, unless you are that horse of course. For another contender, there may be an issue of a recent illness, but again, a rare occurrence across the field.

These contextual factors are hard to capture for any systematic model, and probably forms a kind of 'exception' rule at best. But its hard, and not usually within the realm of traditional factor-based econometrics.

How we Built PunterGPT

From that somewhat technical conversation, emerged the idea of PunterGPT, an application of the language models of ChatGPT and Claude 2. We would use these models to analyse and summarise the nuance and the contextual factors that the market of punters are talking about, and find important. We would rely upon the 'market experts' to determine these, there are no independent statistical tests. What we would use the language model was to aggregate these factors and attempt to sift out the relevant contexts, and use (horse-racing specific) language as a kind of gauge to see relevance.

We researched the most concentrated and context rich content online regarding the race. This was a quick manual task, but we could have used language models themselves to do this search, and identify the sources of input. This was critical so that we understood where our (language) data came from.

In the end, we landed on these three websites:

• https://www.punters.com.au/news/melbourne-cup-2023-preview-20231104/
• https://www.foxsports.com.au/horse-racing/melbourne-cup/melbourne-cup-2023-form-guide-horses-tips-odds-field-analysis-of-every-runner-latest-vauban-gold-trip/news-story/a29ea4b2a0d8f0533b60156fedafec1f
• https://www.ladbrokes.com.au/betting-info/melbourne-cup/melbourne-cup-horses/

This created a document of roughly 24 pages (roughly 1 page per horse) of punter analysis. We then fed this analysis into Claude 2 as input into their model, and identified that element as 'PunterGPT'.

We started working with the following prompt, and used various versions of it as we investigated the punters:

You are punterGPT, an expert system that understands all of the nuance and subtle information around horses and their likely form before a big race. The following is a description from the other punters on the street. Provide the top six horses that you think are most likely to be in excellent form.

For each horse, the description was stunningly thoughtful, balanced and articulate. The reasons being talked about, and the flow of reasoning was very compelling, even as we knew that there was no formal science behind the analysis. (sound familiar?)

We wanted to explore the use of the analysis to provide counter-factual arguments. Tell me why it may not be a good idea to bet on a horse. Something that is often hard to find in punting magazines, unless it is implied.

The analysis once again surprised us. Here was a set of reasons and counter-reasons also drawn from the punter information, and things that could be perceived as both negative or positive.

We were curious as to how we could further use this to make counter-arguments. We focused on the favourite, Vauban, which looked like the horse to beat. (Vauban did not place in the top 3 in the end).

The counter-arguments were stunningly focused and readable. The information is crafted from the inputs is clever, point 3 on Weight is : "He blitzed them with 57.5kg last year but goes up an extra kilogram and the legendary Makybe Diva is the only horse to lump more than 58kg to Cup victory since Think Big in 1975.".

However, is not entirely correct. The original text suggests that two horses had more than 58kg, and one had 57.5kg. Therefore, the three horses qualify as having won carrying at least 57.5kg, not 55.5kg as implied in the summation text. There are no other mentions of historic weight and winning in the text.

Other interesting questions we asked:

• What are the factors that the being considered by the punters in the market?
• What are the factors that are not being considered, but should be?
• What are the contextual factors that are in the following statistical model, but that are not mentioned by the punters as important?
• What is a factor that is being (overly) focused on by the punter community?
• What is something that is implied, but not explicitly written about each horse?

What we learnt

Once again, language models impressed us in their capability to summarise, and interrogate written text. The richness of the text and contextual nature of the information was easy to understand and keep tabs on. As financial services professionals, this reminded us of the barrage of detail and data that we get each day in our inboxes.

• Language models brought us closer to this content, and allowed us to dive in and summarise much more than otherwise. It allowed us to build cases for and against, to look for hidden meaning, and to analyse the market voices themselves.
• For cross-sectional, and often heterogeneous information like market summation, with highly context specific language, we found this type of analysis extremely accessible.
• We learnt that context window length is extremely important, and that attention wasn't always evenly split. We needed to ensure that we controlled not only the data that we used, but its positioning relative to other data.
• We learnt that context can be retained or disappear as the conversation grows longer. This was something always to keep an eye on.

As an exercise in using experimental technology to augment a successful and established investment process, it was a powerful exercise.