The entertainment industry turns to algorithms in search of blockbuster movies

• Technology that creates impressive digital characters
• Benjamin isn’t your typical movie director
• Comparing how different actors affect the bottom line
• Software assesses scripts better than humans
• The Israeli startup Vault looks into the future
• Learning what the audience likes reduces risk for studios
• How Netflix turned Hollywood to big data
• AI is valuable even if it can’t replace the ingenuity of producers

The rise of the entertainment and media (E&M) industry has transformed the way people spend their leisure time. Watching movies, television, and streamed content is one of the favourite activities of millions of people who spare no expense in learning what happens in the latest season of Game of Thrones or which Avenger kills Thanos. This passion translates into big business. In 2018, for instance, global revenue for E&M companies reached $2.1 trillion and is projected to grow to $2.6 trillion by 2023. And while these figures look impressive, the industry faces multiple challenges that make its future uncertain.

For one, entertaining people is risky business. Most movies lose money and can sometimes border on disaster, such as in the case of the Mortal Engines fantasy adventure, whose producers lost over $100 million. Also, Hollywood and other centres of the film industry around the world are becoming less and less creative. Remakes, reboots, franchises, and sequels dominate movie theatres, and out of the 20 highest-grossing films in 2018, only A Quiet Place was based on a completely original concept. Twenty years earlier, more than half of the movies on this list were based on wholly new ideas. And faced with pressure to make more blockbusters and decrease production cost, filmmakers are increasingly turning to artificial intelligence (AI). They hope that smart algorithms can help the industry become more accurate in predicting which movies will flop and which will make money.

Technology that creates impressive digital characters

The E&M industry is no stranger to artificial intelligence tools. The marketing departments of major studios use AI-powered advertising tactics to reach people on social media and other digital channels, streaming providers rely on algorithm-run recommendation systems to help customers select the next movie or show to watch, while advanced software tools help studios to create impressive special effects. So, it’s no wonder that AI spending in the entertainment and media industry was valued at $950 million in 2018, and it’s set to reach $9 billion by 2025.

The US visual effects company Digital Domain, for instance, uses smart algorithms to create some of the most memorable digital characters. When it worked on the Avengers movies, its engineers trained AI on high-resolution scans of Josh Brolin’s face and then used another AI algorithm to apply the resulting face renders onto Thanos’ body. The software was so powerful that it even rendered the smallest wrinkles from the actor’s face, but it’s expensive and out of reach for most of the smaller studios.

Luckily, the US startup Arraiy is developing an AI-powered program that would automate much of the work involved in creating digital effects for movies, television, and games. Instead of specialists spending hours on isolating people and objects filmed in front of a ‘green’ screen and then applying digital effects, algorithms could one day do much of these tasks automatically and in real time.

Benjamin isn’t your typical movie director

But some filmmakers have far more ambitious plans for AI. The Los Angeles-based movie director Oscar Sharp and the software engineer Ross Goodwin have developed a smart algorithm named Benjamin that creates films all by itself. In 2016, the duo debuted their software at the Sci-Fi London film festival, feeding a lot of Sci-Fi scripts into the neural network and telling it to write its own script. The result was a short Sci-Fi film that made absolutely no sense narratively, but it showed just how ‘imaginative’ AI can be.

And in 2019, Benjamin is doing everything alone. It now uses face-swapping and voice-generating technologies to write the script, select scenes, and put sentences together using voice recordings of various actors, compared to just using text. The software runs on the Amazon Web Services cloud and Google’s machine learning network TensorFlow and demonstrates the rapid progress that the engineers achieved in just a few years. But the development of face-swapping techniques raises concerns about their use in creating deepfakes. These manipulated videos, in which the face of one person is put on another’s head, can potentially create chaos, especially during the elections. The actor Thomas Middleditch, who took part in the Benjamin-generated movies, says that every tech starts with positive intentions from the people behind it. “But as with everything that people create, others will come around and infiltrate it with their own mysterious means and intent, especially facial technology,” he concludes.

Comparing how different actors affect the bottom line

Artificial intelligence can also help studios to predict the financial effects of certain decisions. The Los Angeles-based startup Cinelytic runs an AI-powered program that cross-references how much money a movie made with key themes and actors that took part in it. By using these and other datasets, the algorithm enables the user to replace one actor for another and see how that affects box office performance. The company’s CEO, Tobias Queisser, says that the software could, for instance, calculate how replacing Emma Watson as the lead role with Jennifer Lawrence could affect sales in different territories.

But predictive programs work best as a supplementary tool, says Andrea Scarso, the director of the UK-based investment company Ingenious Group. They can validate the thinking of movie executives and suggest certain ideas and approaches that the team wasn’t even considering. He notes that “Having something like Cinelytic, together with our own analytics, proves that [the suggestions] we’re making aren’t just our own mad ideas,” but the ultimate decision is always made by industry experts. This approach is also justified by the fact that algorithms rely on historical data and are unable to account for present or future shifts in culture and customer tastes.

Software assesses scripts better than humans

The Belgium-based tech firm ScriptBook promises even more than Cinelytic. Its software, trained on a dataset of 6,500 existing scripts, requires users to upload a PDF file of a screenplay, and a few minutes later, they’ll receive a report that calculates the MPAA ratings, assesses the emotions of characters, defines the target audience, makes box office predictions, and delivers many other types of information about the movie. The company’s founder, Nadira Azermai, says that movies based on scripts approved by the system have an 84 per cent success rate, three times more than those authorised by humans. She also tested the program using scripts of 32 failed movies produced by Sony Pictures in the period from 2015 to 2017. ScriptBook correctly identified 22 of those movies as box-office failures.

Movie distributors can benefit from the software, too. Instead of relying on gut feeling when buying rights to multiple territories, they can turn to algorithms to assess the script and make data-based decisions. Besides this, ScriptBook improves gender parity in movies by testing whether the script includes “at least two female characters having a conversation that is not about men”, as well as telling directors how many dialogues are between two men, how many between two women, and how many between members of opposite genders. But using ScriptBook doesn’t come cheap. The company charges around $5,000 for customers to insert a single script and generate the report, although it offers discounts for clients who want to analyse multiple movies.

The Israeli startup Vault looks into the future

The Israeli startup Vault managed to solve one of the key problems posed by algorithm-based solutions: the reliance on past data and the lack of insight into future trends. Its predictive analytics and marketing platform enables movie producers to have a sense of trends that will dominate the market six months in advance, as well as how various groups will react to the content. The software makes such predictions by analysing the movie outline, script, trailer, and final cut. Marketing departments will find Vault especially useful, as it helps them identify core concepts that ensure that advertising campaigns are well received by the audience.

The platform’s value lies in its ‘what if generator’ that provides answers to various questions. And to stay on top of future trends, Vault’s algorithms are constantly fed with new data from various sources. The company’s CEO and co-founder, David Stiff, says that they look into podcasts, music, and many other types of content and how they reflect on the trends that will dominate the box office in a few years.

Learning what the audience likes reduces risk for studios

The American film studio 20th Century Fox has also taken a keen interest in AI and has teamed up with Google to learn more about the preferences of moviegoers. The two companies developed an AI tool called Merlin Video that analyses movie trailers and labels various objects and events seen in the footage. Then, it compares this data with labels from other trailers in an attempt to find overlap between the audiences.

For instance, the software analysed Logan, an action movie featuring Hugh Jackman as Wolverine, and returned labels such as facial_hair, vehicle, and beard. In the next step, Merlin compared these labels with other trailers and discovered that fans of various superhero films like Logan, too, which isn’t surprising. But the most valuable insight was that the fans of action movies such as John Wick, dominated by rugged, main male characters, overlap with the Logan audience. This enables 20th Century Fox to learn more about the micro-preferences of their audience, helping producers to know what characters and plots to develop and ensuring that marketing teams emphasise relevant messages in the promotion phase.

How Netflix turned Hollywood to big data

And while a number of studios around the world now use predictive algorithms, many were distrustful not too long ago. A ScriptBook data scientist, Michiel Ruelens, says that “when we started out four years ago, we had meetings with big companies in Hollywood. They were all super sceptical.” But as they saw what AI programs can do and learned how to use them, movie executives began to increasingly trust this technology.

However, Netflix was probably the single most important factor that turned Hollywood to big data. This content streaming giant boasts an impressive recommendation algorithm that’s alone worth $1 billion a year. It collects user data, helping the company to serve relevant recommendations and decide which original movies and shows to produce. And in 2018 alone, Netflix reported $15.8 billion in revenue, demonstrating the power of a data-driven approach to content production and distribution.

AI is valuable even if it can’t replace the ingenuity of producers

Millions of people around the world are willing to spend money to watch exciting movies and shows. But producing profitable content is no easy task, which is why the entertainment industry is embracing smart algorithms. Instead of relying solely on gut feeling and the experience of senior executives, companies can now take data-based recommendations into account when selecting, producing, and promoting their movies. AI also helps designers to create stunning visual effects and impress their audience. And while technology can’t replace the ingenuity of producers and scriptwriters, it can surely do everyone a favour and make sure that we see fewer bad movies and more of those that will be worth our time and money.

This article first appeared on the Richard van Hooijdonk blog at https://richardvanhooijdonk.com/blog/en/the-entertainment-industry-turns-to-algorithms-in-search-of-blockbuster-movies/