AI Subtitling for Short-Form Video via an Adobe Panel

VRT Sandbox facilitates the screening, testing, coaching, trial, and funding of start-ups. Annually, the program extends its support to international scale-ups, as well as growing and emerging technology companies from various European countries. Collaborating with VRT, these entities develop innovative media tech or content. In 2022, VRT Sandbox prioritized four themes through the STADIEM project, namely synthetic voices, audience engagement, content management, and automatic subtitling. VRT conducted interviews with each participating start-up from the STADIEM project to gain insight into their tool's capabilities and how VRT can utilize them.

Today we talk about automatic subtitling with Maarten Verwaest, founder of Limecraft.

What is Limecraft?

Maarten: 'Limecraft is a scale-up that provides media makers with a platform to enhance their production processes. In collaboration with VRT, Limecraft devoted a year to developing automatic subtitling for short-format videos, which are brief videos. The goal was to design a user-friendly application for media makers. The team leveraged speech recognition and natural language processing to create a tool that enables editors to automatically subtitle videos. Editors can seamlessly add subtitles to their videos through a few simple clicks after finishing the editing process in Adobe Premiere Pro. This process eliminates the need for manual subtitling, allowing editors to obtain high-quality subtitles in just a minute. The editor can focus on a quality check instead of spending an hour manually subtitling. By streamlining repetitive tasks, the editor can concentrate on what truly matters: storytelling.'

Why is Limecraft important for VRT?

Maarten: 'In line with its digital-first strategy, VRT seeks concrete, user-friendly and digital applications that optimize media makers' time. Despite the various forms of speech recognition technology available, it remains a challenge to implement them in practice. VRT ensures accessibility for all videos by providing subtitles. However, the traditional method of subtitling involves subtitlers who require an extended turnaround time for programs or documentaries. As a result, editors must manually subtitle short videos, which can be time-consuming and challenging, particularly for those who lack subtitling expertise. Consequently, VRT sought a solution to assist editors and enhance the quality of their work.'

What did Limecraft test with VRT?

Maarten: 'A diverse range of VRT content, spanning from news to entertainment, was subjected to testing. Short-form videos proved to be the most suitable for subtitling, as they typically involve only one language and can be easily edited post-production without specialized software. With the aid of artificial intelligence, such videos can be subtitled quickly and efficiently.'

To be as realistic as possible and take into account the short time to deliver results, VRT narrowed down the use case, exploring the different options that specifically apply to short-form video.?

VRT is looking for answers to the following three key questions:

• Is it feasible in the first place to use AI for automatically creating subtitles, whereby video creators only have to do some spot checks?
• Overall, is it possible to create high-quality subtitles for short-form videos at a substantially lower cost in less time?
• Doing so, is it possible to increase the probability of delivering videos with high-quality subtitles?

What are the obstacles Limecraft faces?

Maarten: 'Automatic subtitling faces challenges with sloppy language or poor recording quality. Additionally, videos with multiple languages may not be translated with 100% accuracy by the model. Thus, automatic subtitling performs best for videos with a single language. It has been observed that news and documentary formats benefit the most from the time-saving feature, while fiction is more challenging to process automatically. Ultimately, users must familiarize themselves with automatic subtitling to determine the types of material that yield optimal results.'

Implementation

1. Recycling of subtitles originally created for live content

For this part of the Proof of Concept, VRT supplied a corpus of 20 factual programs that have been live subtitled using?respeak. The resulting subtitles are of reasonably good quality (+/- 1% of the subtitles may contain wrong interpretations), but they have a variable delay of +/-10 seconds. The key question is to what extent these subtitles can be re-aligned automatically.

To illustrate the experiment, Limecraft uses the example of?De Afspraak, in which a panel discusses current affairs live for 50 minutes. They have the result of the live subtitling (respeak + correction) at their disposal as an STL file, containing 750 subtitles with a timing offset of roughly 10 seconds.

2. Using AI transcription and subtitling as a proxy

In cases where no pre-existing texts or subtitles are available, the next best alternative is using AI transcription and the subtitling spotting algorithm. For content with suitable audio (factual programming, a trained voice, grammatically correct language), we know from experience +/- 90% of the subtitles match the quality of otherwise manually produced subtitles, saving 75% of the total turn-around time.

In the clip below, they compare the result of ASR and AI subtitling 36 months ago (in the middle of the screen) with state-of-the-art processing (bottom of the screen).

3. Make available the results to the editor using an Adobe Panel

The final and arguably crucial task is ensuring the accessibility of the technologies to video editors who may not be proficient as professional subtitle editors. In consideration of VRT's specific needs, an Adobe Panel has been created to initiate AI transcription and subtitling, and subsequently import the completed results into Adobe Premiere. This approach provides video editors with an automated process, eliminating the need to leave their workspace. The output has a high accuracy of over 95%, making it suitable for finishing in Adobe.

What about STADIEM?

STADIEM (Startup Driven Innovation in European Media) is a program that facilitates the development of next-generation media solutions by bringing together startups, scale-ups, investors and media organizations through its piloting and acceleration program.

We are now going to the next phase: The Integrate Phase. 12 scaleups are selected to continue to the next step in this program, Limecraft is one of them. This phase focuses internal testing and evaluation of the solution and preparing a strong business case.

Want to know more about our next-generation acceleration program STADIEM?Take a look on https://www.stadiem.eu.