Balancing Act: The Promise and Perils of Generative AI in Film and TV

Generative AI is evolving and advancing at warp speed, and the pace of change is both staggering and unnerving. Its influence and implications have swept across the creative and knowledge-based professions, including the film and television industry. Those implications were brought into sharp relief recently with the advent of Sora AI , a generative AI model developed by OpenAI that specializes in text-to-video generation.

The results of Sora AI's advanced video generation capabilities are striking, featuring vivid scenery and landscapes of stunning richness and incredible realism. The model has, unsurprisingly, taken the film and television industry by storm and shaken the major Hollywood studios.

The upshot is a powerful demonstration of generative AI’s potency as a game-changer for many of the film industry’s core functions: From scriptwriting and pre-visualization to content localization and sound editing, industry leaders can harness AI tools to enhance the creative process, introduce new efficiencies into the video production workflow, and innovate more than ever before. Let's explore some of the most compelling generative AI use cases for the film and TV industry, as well at the potential drawbacks keeping industry leaders up at night.

The Promise of Generative AI for the Film Industry

Sound Editing and Graphics

Generative AI can help with voiceovers, sound effects and themes . With the help of AI voice cloning, filmmakers can continue using an actor’s voice after they have, for example, aged or passed away. AI also generates sound effects appropriate to the given video materials, like footsteps or ambient environments. AI music generation tools like Stable Audio (Stability AI) can come up with film scores or TV themes using a wide range of music genres, instruments and vocals.

Another area where generative AI can help is graphics. Phil Siegel , founder of the nonprofit Center For Advanced Preparedness and Threat Response Simulation , thinks that tools like Sora AI will reduce the time and cost of making movies. It’s often required for a whole team to travel to a destination just to capture the lead-ins for crucial scenes, but with the help of generative AI, some movies might use this technology to generate master or establishing shots instead at a fraction of the cost. This kind of tool also helps editors add or subtract content during editing phases, which would otherwise be a more complicated process.

Content Localization

Generative speech tools can also be used for audiovisual content localization. Trained on large quantities of real speech data, AI models can replicate or closely match specific voices, modify a voice's tone or words spoken, and create entirely new synthetic voices.

Using AI technology for dubbing can help film and TV professionals enable rich performances that sound like the original actor or performer across language versions. At the same time, using AI tools makes dubbing more financially accessible and can potentially unlock distribution for movies and TV programming into new markets.

One of the interesting examples of AI-powered apps in the content localization area is Deepdub, used by Hollywood studios and major streaming services. Deepdub offers audio dubbing into different languages, as well as auto-audio splitting, dialogue isolation controls, plus other lip movement and timing sync options. It also has an "accent control" feature that allows for instant tuning of the performers’ accents to be more in line with the region where video content will be presented.

Scriptwriting

In AI-assisted scriptwriting, "assisted" is the operative word. According to the deal that ended the US writers' strike in 2023, studios "cannot use AI to author scripts or to edit scripts that have already been written by a writer." The agreement also prevents studios from treating AI-generated content as "source material," like a novel or a stage play that screenwriters could adapt for a lower fee and less credit than a fully original script.

While there are some restrictions, technically AI tools like SoraAI can help automate and diversify scripts. For instance, by analyzing vast datasets of genre trends, viewer preferences, and narrative success factors, Sora AI can generate compelling story arcs and dialogues. It can even suggest casting options to match the envisioned characters by analyzing thousands of actor profiles, performances, and viewer reactions.

Nonetheless, today’s Large Language Models (LLMs) struggle to generate a truly coherent and production-ready output in the manner of a screenplay without human involvement. As the capabilities of LLMs further advance, writers and studios will remain central to the process of leveraging AI-powered tools for scriptwriting in the best ways through experimentation, testing, and evaluation.

Streamlining Production Processes

Generative video AI also aids filmmakers and video professionals in streamlining their production workflows. Using AI-generated images and videos in production is problematic due to legal copyright restrictions. However, AI technology can be used in pre-visualization stages, such as generating 2D images, 3D models, moodboards, storyboards, and animatics that display a project vision, style, scope, and narrative arc. In this context, AI tools can add value by accelerating the creative process, feedback rounds, and automating repetitive tasks.

The Perils of Generative AI in the Film Industry

Earlier in this article, I raised some caveats about the application of generative AI in the film industry, such as the unfettered use of AI in scriptwriting and the legal limitations on using images and videos generated by AI in production.

A few other concerns to ponder:

• Digital doubles: Creating replicas of actors is one of generative AI's many capabilities, but there's been plenty of dispute around the ethical basis of using actor clones. In 2023, actors went on strike, and argued that their replicas were being created without their consent, and could potentially leave them without any compensation. In November 2023, the strike ended with an agreement on AI usage, which included the need to acquire actors' consent to create their digital replicas as well as the guarantee that whether it's an actor or their digital double onscreen, the actor gets paid.
• Job displacement: With generative AI being explored across so many dimensions, the prospect of mass job displacement threatens not only actors. Roles in nearly every domain of the film industry could be either partly or entirely replaced by AI technology in the future. As long-time entertainment industry executive Gilbert Galvan lamented "in the next three to five years, this technology might displace roles on creative and production teams."
• Intellectual Property (IP) infringement and data privacy: US entertainment industry professionals are deeply concerned about how generative AI technology might impact their roles, companies and the industry. A majority of industry professionals expressed being either "very" or "somewhat" concerned about several potential negative outcomes of generative AI, including IP infringement across copyright, trademark, and right of publicity; job losses resulting from employers' use of generative AI; and issues related to data privacy.

Final thoughts

Given the real-world use cases for the film and TV industry outlined above, generative AI has clear potential to revolutionize the film and TV industry. At the same time, there are legitimate fears about the implications of employing generative AI without adequate constraints or attention given to the ethical and legal concerns. With careful strategic thinking, planning and coordination in how generative AI should be best deployed and used by industry leaders and insiders, generative AI tools can help filmmakers to improve their creative process, make production workflows more efficient, reduce costs, and penetrate new markets.

If you're looking to embrace the power of AI, DataArt can be your partner for progress and help you stand out and deliver tangible business value. At DataArt, we are committed to helping our clients find innovative ways to solve their business problems and challenges with the help of AI, and we've already completed 40+ generative AI POC projects in just the last six months.

If you want to talk about our AI expertise and capabilities and how we can help, please DM our expert, Doron Fagelson.

Originally published here .