Behind the Scenes: Music Video Breakdown

As we see new AI video generations popping up every day, with fresh tools pushing the boundaries with shared tips, tricks, and 'how-to' guides here and elsewhere, it feels like AI is advancing at an incredible pace. What used to take months now happens in weeks, and weeks are turning into days.

While it's still fresh, I wanted to share my 'two cents' about the music video I posted a few days ago and how each of AI tools played a role in generating the shots incorporated in the video production.

I won't delve much into the song itself (I might do a separate post about songwriting and ideation) and what inspired me to create it in this particular downtempo/alternative genre, often recognized as a hallmark of the '90s. The main point is that the lyrics are not AI-generated by Udio or ChatGPT. Instead, I wrote them myself, using ChatGPT only for listing rhyming words and checking grammar and meter, with minimal alterations from ChatGPT.

I'm not exactly sure how the idea of using subways, trains, and tunnels came about. I was already experimenting with some subway shots in MidJourney. Plus, the music, with its repetitive beat and oriental chanting, reminded me of the '90s MTV era. Back then, many videos used collage shots to tell a story that matched the music or lyrics, or both.

I'm also including a reel of generated shots that didn't make it into the final video. Here’s why:

• The video was good enough and even made it into the mix during the first versions, but for aesthetic reasons, it had to be edited out.
• The video had great composition and a solid character, but the camera movements were too unpredictable.
• In what seemed like a good-quality video, the character started acting unpredictably, moving forward, backward, or through walls.
• Character face morphing: as soon as the angle and position of the head changed, the character looked like a different person.
• Reflections: with so many parts of the background featuring subway train car windows, reflections of the character's faces were inevitable. Controlling these reflections was tough since they didn't mimic the character's movements most of the time.
• The background and people in the background acted unpredictably, even when the main character was in a perfect state.
• For this particular video, using a subway train as the main motif and mentioning it in the prompts to either Luma or Runway mostly resulted in videos that were shaking too much.

While generating these shots, certain features of the overall video started to emerge. Seeing this potential as a narrative device, I decided to incorporate and keep it as the basic overlay of the whole music video. There are three main presets I kept within the video:

1. Cold Green (Almost Blue) Neon Light: Visible in most of the scenes, this element started as an attribute of a few originally generated videos. It required a lot of color correction to achieve a consistent look throughout the entire video during editing.
2. Winter Clothing: The characters are mostly wearing puffy winter jackets and elegant winter coats. This decision was made to enhance the overall cold atmosphere, complementing the green neon glow.
3. Serious Faces: The characters' faces needed to look serious and express worry, adding to the mood and tone of the video.

Runway Gen-3 Alpha

I will go through some of the important facts about each video tool as pro and con list. Let's see some of the pros about Runway Gen-3 Alpha:

• Creates high-fidelity video shots with significant improvements in motion and consistency over previous models (Gen-2)
• Utilizes cutting-edge infrastructure designed for large-scale multimodal training, pioneering steps towards comprehensive world models
• Suitable for various fields like film production, visual effects, and immersive storytelling, enhancing visual storytelling capabilities
• Easy-to-use prompts guide the creation process and allow exploration of creative ideas with AI-generated videos
• Introduces innovative techniques and technologies in AI video generation, encouraging experimentation and innovation in video production

Here are some limitations or cons that I'm confident won't remain on this list for long:

• Supports only text-to-video generation, making it challenging to maintain character and shot consistency.
• Similar shots can be created, but transitions are tricky due to the lack of control over the final videos, even with precise prompts.
• Characters often act unnaturally, displaying strange movements, morphing faces, and erratic camera trajectories, disrupting video flow.
• The cost can add up quickly, with the flexible credit system becoming expensive for extensive projects, especially if results don't align perfectly with your vision.

Luma Dream Machine

List of pros for Luma Dream Machine:

• Dream Machine, created by Luma Labs, is a text-to-video and image-to-video model known for its realistic motion capture.
• Allows users to upload images created with tools like MidJourney or Leonardo for greater control over the video output.
• Creates five-minute videos.
• Extends the duration by adding more seconds to an already generated video.

Luma Dream Machine certainly recognized the importance of their image-to-video model, so a few weeks after the launch, they released an important update where users can:

• Upload an initial image as the first shot.
• Upload a second image as the final shot.
• More control over the final product

Cons of Luma Dream Machine:

• Achieving the desired shot often requires multiple attempts.
• Even with precise prompts, the tool can produce unexpected results.
• Camera movements may go off track.
• Characters can display unnatural morphing.
• Video quality can vary, with occasional surprising movements and inconsistencies in character actions.

Rigid Pricing Plan Cons:

• Beside Free Plan, there are also Standard, Plus, Pro, and Premier plans offer different pricing and credit allocations.
• Once you exhaust the credits in your current plan, the only way to get more credits within the same month is to upgrade to a higher plan.
• The system lacks flexibility, forcing users to either wait until the next month or switch to a more expensive plan.
• A more flexible option for purchasing additional credits as needed would greatly enhance the user experience.

Walking Away | Music Video

AI's rapid advancements are transforming video production, opening up amazing creative opportunities. Today's innovations quickly become tomorrow's standards as technology evolves at a remarkable pace. Tools like Runway Gen3 Alpha and Luma Dream Machine show just how AI can enhance storytelling and visual effects. I'm excited to keep exploring these tools and have some new projects lined up for next month. Stay tuned for more updates!

#AI #VideoProduction #AIVideo #RunwayGen3 #LumaDreamMachine #AIStorytelling #CreativeTech #Innovation #FutureOfVideo #TechAdvancements #AIArt #DigitalCreativity #VisualEffects #NextGenTech #StayTuned #Midjourney