When a Crystal Ball Isn’t Enough to Make You Rich

By Victor Haghani, James White and Jerry Bell

Estimated reading time: 6 minutes

Introduction: Back to the Future

In the 1989 blockbuster?Back to the Future II, time travel enables Michael J. Fox’s nemesis, Biff, to become a gazillionaire by bringing an almanac with sports match outcomes back from the future. We thought it might be instructive, and certainly entertaining, to make a less fanciful version of this dream a reality – for a few lucky people.

In November 2023, we ran an in-person, proctored experiment involving 118 young adults trained in finance. We called the experiment “The Crystal Ball Challenge.” We gave each participant $50 and the opportunity to grow that stake by trading in the S&P 500 index and 30-year US Treasury bonds with the information on the front page of the Wall Street Journal (WSJ)?one day in advance,?but with stock and bond price data blacked out. The game covered 15 days, one day for each year from 2008 to 2022.

You can play this game for yourself here:?Crystal Ball Trading Challenge?– though without the pecuniary component. As of the time of writing, over 1,500 people have tested their skill and luck by playing the game on our website.

Summary of results

The players in the proctored experiment did not do very well, despite having the front page of the newspaper 36 hours ahead of time. About half of them lost money,?and one in six actually went bust.?The average payout was just $51.62 (a gain of just 3.2%), which is statistically indistinguishable from breaking even. The poor results were a product of: 1) not guessing the direction of stocks and bonds very well, and 2) poor trade-sizing. The players guessed the direction of stocks and bonds correctly on just 51.5% of the roughly 2,000 trades they made. They guessed the direction of bonds correctly 56% of the time, but bet less of their capital on bonds than on stocks (if you’re planning a career as a proprietary macro trader, consider putting your focus on bonds).

Perhaps the front page of the WSJ isn’t a particularly clear crystal ball, or our players weren’t very adept at reading it. As former Goldman Sachs CEO Lloyd Blankfein reminded us in a widely-circulated tweet, sometimes the markets don’t react to the news as even seasoned experts expect – an important lesson all successful traders learn, eventually.

It didn’t help that the players also did not seem to know how to size their bets well. On eight of the 30 trading opportunities,2?the players in aggregate displayed 2-to-1 odds of being correct in their bets, but they did not bet more heavily on those occasions. Overall, they did not display trade-sizing that bore any relation to their propensity to guess the price moves of stocks or bonds correctly.

Many of the players used excessive leverage relative to their exhibited edge in guessing market direction. On about 30% of the total number of days on which players traded, they used leverage of greater than 20x capital. On 4% of the total occasions, they used leverage of 60x or higher, which carried a very high probability of being wiped out if they guessed wrong. In sum, there was little discernible logic or rationale to their trade-sizing decisions.

See Appendix I for a detailed analysis of player results.

Perhaps this excessive risk-taking by some of the players is partially explained by the finding that most investors tend to overestimate the predictive value of news on market outcomes. For example, a recent survey of 11,000 investors by Andre et al.3?found that about 70% of investors (but not finance academics) believed that stale, four-week-old, good (or bad) news was predictive of high (or low) future stock returns.

However, our sample of 118 staked and proctored players did better than the roughly 1,500 people who have played the game for fun on our website. The median outcome among these players was a loss of about 30% of their capital. Only 40% finished with a profit, and 36% went bust.

We were tickled to see that six players devoted themselves to achieving the maximum possible payout, growing their initial wealth 70,575-fold. They did this by repeatedly playing the game to see what stocks and bonds did on each day, and then using that information to put up the perfect score by correctly betting the maximum size on each trade. We were elated to see our game spark so much passion in some players!

Some of the world’s best traders show how to do better

We invited five seasoned and successful macro traders – four men, one woman – to play the game, with markedly better results. This was a very select group of traders: head of trading at a top-five US bank, founder of a top-ten macro hedge fund, senior trader at a top-ten macro fund, former senior government bond trader at top-three US primary dealer, and former senior Jane Street trader.

These players all finished with gains. On average, they grew their starting wealth by 130%, with a median gain of 60%. All of the players were selective and highly variable in their trade-sizing. They did not bet at all on about 1/3 of the trading opportunities, but bet big on days when they presumably felt confident in the impact of the news on stock or bond prices.4

These veteran traders predicted the direction of the markets significantly better than our 118 younger, less experienced participants (63% vs 51.5%), but mostly we ascribe the dramatically different results to the much more rational trade-sizing displayed by the experienced traders. One important conclusion we reach is that there is little value in this crystal ball?without sensible trade-sizing.

Motivations

In addition to our curiosity in testing Taleb’s hypothesis, we had four further motivations for conducting this experiment:

1. We are deeply interested in learning how people approach the sizing of attractive investment opportunities, having researched and written extensively on this topic. In 2016, we conducted an experiment (also involving financial rewards) where we invited participants to bet on a digital coin flip that was programmed to have a 60% probability of landing on heads, and published our findings in “Betting on a Biased Coin.” We recently wrote a book,?The Missing Billionaires: A Guide to Better Financial Decisions, that is focused on investment sizing.
2. We wanted to quantify the value of macro-economic information. How often would people guess the direction of markets from the information on the front page of the WSJ?
3. We hoped that the tool we developed for the experiment could be productively used to educate and train professional risk-takers.
4. It was going to be a lot of fun!

The Game

Over 90% of the participants were in graduate programs in finance or MBA programs with finance modules at four east coast US universities with low admission rates. The participants were not told in advance that they’d be invited to participate in this experiment. Any who did not want to participate were allowed to leave (though no one did).

Here’s how we explained the rules of the game:

We are giving you $50 to play our “Crystal Ball” game. The object is to see how well you can do trading stocks and bonds if you know the news from the front page of the WSJ one day in advance. In other words, you’ll be in that dreamed-of position of being a trader who “knows the future.” For example, you will be shown the front page of the WSJ for a Wednesday, and be able to take a long or short position in the stock market and in the bond market at prices prevailing at Monday’s close (that is, two days earlier). Your trades will be liquidated at Tuesday’s closing prices. Note, on each front page we’ve blacked out anything that tells you explicitly what market prices actually did that day – leaving that information in would make this game too easy and no fun at all!

You will be trading the S&P 500 stock market index and a 30-year US Treasury bond futures contract, and you can use as much leverage as you’d like to, up to 50x. Use the sliders to choose the positions you want and then click the “Trade” button. Remember that for the 30-year Treasury bonds, prices go down when yields go up, and you are trading on price.

You are starting off with $50 of bankroll, and we will pay you however much this has grown to, or shrunk to, with a maximum payout of $100. You will have 45 minutes to play the game.

We have not chosen these days to try to trick you – they are randomly chosen. You will be able to trade on 15 different days, once per year over the past 15 years. The days will be presented to you in a randomized order. We’ve chosen these days randomly from a set of days where one third of them are days of employment reports, one third from days of Fed announcements, and the other third purely randomly, all taken from days that are in the top half of days ranked by overall market volatility. You can use the “Skip” button to skip any day you don’t feel like trading, and you can trade stocks, bonds, or both, each day. You can use the “Finish” button if you want to stop before being presented with all 15 days. Leverage of 1x means your position size is equal to your capital size. There are no transaction costs or overnight financing costs or rebates on your trades.

Good luck, and have fun – you may never have this opportunity again!

Below are two pictures of the screens that players engaged with. In the first screen, the player can expand the picture of the front page of the WSJ on the left to be able to read it more clearly, then revert to the trading page.

Then, after clicking “Trade” the result of the trade is revealed, showing the market move in stocks and bonds, and the resultant profit or loss. The player’s bankroll is expressed based off of a starting value of $1 million, but the players understood that their payout would be $50 times the ending wealth divided by $1 million, with a minimum of $0 and a maximum of $100.

All the front pages can be seen here, in chronological order:?https://elmwealth.com/crystal-ball-gallery/

Conclusion

Was Taleb correct in his conjecture that “If you give an investor the next day’s news 24 hours in advance, he would go bust in less than a year”? While our experiment didn’t test his statement precisely – we only gave players 15 days of front pages, players were risking just $100 in the game, etc. – by and large we think Taleb is right. His counterintuitive proposition is both insightful and instructive.

The financial industry is replete with individuals and organizations constantly working to develop their own proprietary crystal balls. We hope that the experiment and results described herein convince crystal ball makers that sensible investment-sizing is essential to realizing the value of what they are trying to build.

The poor aggregate showing of our 118 financially-trained participants highlights the importance of educating young, aspiring finance industry professionals in decision-making under uncertainty, and particularly the theory and art of investment-sizing. We hope our Crystal Ball game will be a helpful tool – or a prototype for a better one – that educators and financial firms can use to teach these concepts and skills. Perhaps it may even become part of the hedge fund boot camp training programs at Citadel, Point72, Balyasny, and Jane Street that have been in the news recently.5?The uniformly positive results of the five experienced macro traders we invited to play the game suggest that there are teachable skills involved in successful discretionary investing.

Perhaps Matt Levine foresaw the results of our experiment with his article titled: “Knowing the Future Isn’t That Helpful.”6?He describes a delightful academic study that analyzed the trading results of a cartel of investors with an excellent, albeit illicit, crystal ball.7?The traders bought earnings announcements before they were released from an international hacker group that illegally obtained access to the servers of three commercial newswire companies. These traders were sophisticated and their crystal ball was gem quality, but their batting average was far from perfect – though it was still good enough to make a decent return on their capital…before they were caught by the SEC!

Most stories involving people seeing into the future, like that of the trading cartel above, don’t have “happily ever after” endings. There are usually unintended consequences that come with perfect prescience – a reminder that even prophets can’t escape risk and uncertainty. The best we mortals can do is make our decisions with a framework that explicitly accounts for the presence of risk in just about every big choice we face.

If you haven’t already, you can play the game here:?Crystal Ball Trading Challenge

For a detailed analysis of the player results and our assessment of the value of a Crystal Ball, check out the appendix on our website: https://elmwealth.com/crystal-ball/