Reborn - Creating a Short Video with the Runway and Midjourney Generative AI Tools in 10 Hours

I have long been curious about the text-to-video solution's outcome in the generative AI field, where we generate video based on?text instructions (prompts) with the help of artificial intelligence. Runway's promising Gen2 product has been available to the public since June. The text-to-video solution can also be found among the features, so I was very eager to test what it can do.

Originally, I wanted to create a short video in which I use some generative AI solution for every element, but in the end I only managed to realize this partly, I will explain later why.

Curious about the process? Let me walk you through it.

1.PREPARATION/STORY/CONTEXT

When using generative AI, it's crucial to provide clear context and input to achieve the desired outcome. Knowing where you want to go and having a clear vision in mind is key to successful collaboration between humans and AI. In this case, I conceived a story that I wanted to bring to life. It revolves around a man living in a closed, dark apartment, isolated from the world. He yearns to break free from the oppressive space. One day, as sunlight streams through the window and caresses his face, he decides to step out of his comfort zone and discover the beauty of the outside world. Symbolically, this story represents both the confinement and liberation experienced during the COVID period, as well as the struggle of individuals trapped in their own minds, determining whether they will make changes, step out of their comfort zones, and embrace new experiences. This served as the foundation for my work.

2. SKETCH/STORYBOARD - MIDJOURNEY

Within the Runway program, there are multiple options for creating videos. I chose the method of providing a textual prompt and an additional reference image to ensure the final result closely matched my envisioned concept. By utilizing Midjourney, a program I am familiar with, I quickly achieved a visual solution that aligned with my mental image. I paired this image with the textual prompt and input it into Runway. In total, I created 34 Midjourney images, which can be seen as storyboard frames. Midjourney greatly facilitated visualizing the imagined world and atmosphere during this type of project. It is an invaluable tool for creative professionals during the preparatory phase.

3. IMPLEMENTATION - RUNWAY

Using Runway, I combined the images and prompts to generate a total of 121 video units, out of which 39 made it into the final material. The unused units were excluded due to unsatisfactory results. The most significant challenge arose when highlighting close-up body parts, such as tying shoelaces, grasping a doorknob, or closing eyes. Runway struggled with these details, but after several attempts, I found a solution that made it into the video.

Here are my observations regarding the Runway program:

• The visual quality is still far from optimal, with grainy and blurry images. However, considering the rapid progress of image-generating programs, we can expect significant improvements within the next 1-2 years as the program continues to learn and refine based on the growing dataset.
• Body parts, especially when emphasizing specific details (hands, eyes, mouth), pose challenges for Runway.
• Lighting effects are surprisingly impressive; I was often amazed by the results.
• Prompt interpretation is less refined compared to Midjourney, indicating the program's need for further learning and development.
• The free usage limit is quickly exhausted, and I even had to replenish my basic package due to credit consumption. Experimenters should keep this in mind.
• The program generates blocks of up to 4 seconds, which was sufficient in my case and even excessive at times. However, this limitation may impact other projects.
• The frame rate within the 4-second blocks is low, resulting in choppy final output.
• Matching styles between video blocks is not yet possible. While the male character's face resembles the one depicted in various video blocks, differences are noticeable. The same applies to the overall style, ranging from animation-like to realistic. This aspect is less controllable during the prompt generation phase.

Overall, I believe Runway is heading in the right direction. It's worth paying attention to its development, as I anticipate it will be capable of remarkable achievements within 1-2 years as the technology refines.

4. MUSIC

Initially, I planned to create the music using an existing generative AI solution. I tried several products on the market, but none met my standards. Therefore, I opted not to employ AI for the music and instead purchased two tracks from Uppbeat, which you can hear in the video.

5. POST-PRODUCTION - PREMIER PRO

Once I had the appropriate video units and selected the music, I used Premiere Pro for editing. Although Runway also offers editing capabilities, I preferred using Premiere Pro due to my familiarity with the software and its faster workflow. Skilled video editors could refine Runway's output even further, but as video editing is not my expertise, I didn't spend much time on this aspect.

In conclusion, I believe Runway is an incredibly useful tool on the product/creative side. It serves prototyping and testing purposes, allowing for quick execution of ideas and concepts (even hundreds of them) within a short timeframe. Although it still has room to grow before achieving impeccable quality, I believe that future improvements are not far off.

#runway #runwaygen2 #midjourney #midjourneyai #generativeai