Creating Short Video Animations using Amazon Bedrock, Stability AI, Photoshop, and RunwayML

Learn to create short animations from a single image using Amazon Bedrock, Stability AI’s SDXL image generation model, Adobe Photoshop, and RunwayML

In this post, we will learn the process of creating short video animations from a single image. We will start with a single image, create multiple variations of the image using Amazon Bedrock and Stability AI’s SDXL 1.0 image generation model, compose select image variations in Adobe Photoshop, and finally, use RunwayML’s Gen-2 Text/Image-to-Video feature to generate a short video from the composed image and a text prompt. Follow along as we break down each step, offering insights into how to leverage these tools effectively for your own projects.

Pre-Production

Base Image

You can choose a single image to start with. Since I was creating an abstract video, I chose an abstract geometric design with no people or words (characters, type, lettering). For this post, I licensed a high-quality vector artwork from Shutterstock . Shutterstock allows you to download an Adobe Illustrator vector file, which may be resized without losing quality, as well as a large JPEG copy of the vector artwork. I started with the supplied 3,508 x 2,000-pixel JPEG file for this post. Be sure to carefully read the licensing agreement from content providers, especially if you plan to use content for Generative AI purposes.

Generating Image Variations

Next, using Stability AI’s SDXL 1.0 model available on Amazon Bedrock, we can quickly and easily create dozens of image-to-image variations using different inference parameters and positive and negative prompts. In a recent video , I detailed how to accomplish this and provided all the code necessary to generate image-to-image variations with Amazon Bedrock on GitHub .

The GitHub repository contains a Jupyter Notebook to help you generate multiple image variations. Below is a snippet of the code I used for this post from the notebook.

# Author: Gary A. Stafford
# Date: 2024-03-28
# Modified version of AWS example code

import random

# Amazon Bedrock Model ID used throughout this notebook
MODEL_ID = "stability.stable-diffusion-xl-v1"

# Location of source image and destination of variations
SOURCE_IMAGES = "./content/source_images"
GENERATED_IMAGES = "./content/generated_images"
SOURCE_IMAGE = f"{SOURCE_IMAGES}/<your_image>.jpg"

# Your prompts
POSITIVE_PROMPT = "(purple:1.5), abstract lines and dots connect background, technology connection digital data and big data concept, best quality, ultra-detailed, masterpiece, high-res, 8k"

NEGATIVE_PROMPT = "(blue:1.5), text, words, letters, type, characters, number, writing, worst quality, low quality, normal quality, low-res, grayscale"

# Loop through ten random seeds and generate image variations
for i in range(0, 10):
    seed = random.randrange(0, 4294967295)
    print(f"Random seed: {seed}")

    imageToImageRequest = ImageToImageRequest(
        image_width=<your_image_width>,
        image_height=<your_image_height>,
        positive_prompt=POSITIVE_PROMPT,
        negative_prompt=NEGATIVE_PROMPT,
        init_image_mode="IMAGE_STRENGTH",
        image_strength=0.2,
        cfg_scale=25,
        clip_guidance_preset="SLOWEST",
        sampler="K_DPMPP_2M",
        samples=1,
        seed=seed,
        steps=30,
        style_preset=StylesPresets.DIGITAL_ART.value,
        extras=None,
    )

    image_to_image_request(
        imageToImageRequest,
        SOURCE_IMAGE,
        GENERATED_IMAGES,
    )        

I generated my image variations at a reduced size but a similar size ratio to the original image, at 1,344 x 768 pixels, to stay within the size limitations of SDXL 1.0 with Bedrock of 1,048,576 pixels.

To achieve the look I wanted, I chose a low image_strength of 0.2, which results in images that are very different from the source image. According to the AWS documentation , “image_strength — (Optional) Determines how much influence the source image in init_image has on the diffusion process. Values close to 1 yield images very similar to the source image. Values close to 0 yield images very different than the source image.”

To achieve the look I wanted, I chose a high cfg_scale of 25, resulting in images closer to my prompt. Again, according to AWS , “cfg_scale — (Optional) Determines how much the final image portrays the prompt. Use a lower number to increase randomness in the generation.”

Lastly, I used positive and negative prompts to change the color from blue to purple. The positive prompt also uses the original description of the source artwork from Shutterstock.

• Positive prompt: “(purple:1.5), abstract lines and dots connect background, technology connection digital data and big data concept, best quality, ultra-detailed, masterpiece, high-res, 8k.”
• Negative prompt: “(blue:1.5), text, words, letters, type, characters, number, writing, worst quality, low quality, normal quality, low-res, grayscale.”

The variations, compared to the original, were fantastic.

Amazon Bedrock Image Playground

You can use the Amazon Bedrock Image Playground to generate image-to-image variations as an alternative to writing code. However, you will need to create variations one at a time, and access to inference parameters is limited in the Playground compared to using the API via SDK or CLI. You lose creative controls, such as style_preset and image_strength.

Image Composition

Once you find several image variations that you like, you can use image editing applications such as GIMP (GNU Image Manipulation Program) or Adobe Photoshop to manipulate the images to your liking. After finding several similar image variations I liked, I used Adobe Photoshop to manipulate and combine the images into a more pleasing composition.

Production

Video Generation

There are several online services and open-source software for generating videos from text and image(s). For this post, I uploaded the final image to RunwayML . Using RunwayML’s Gen2 Text/Image-to-Video tool, I composed several short 4-second video variations (different seeds) from the uploaded image and a positive prompt.

With RunwayML’s Text/Image-to-Video tool, you have fine-grain control over different aspects of the animation, such as camera motion (pan, tilt, roll, zoom), frame interpolation, and upscaling. You also have the option to extend the 4-second video by an additional 4 seconds, which you can repeat to create longer videos.

Post-Production

Extending the Video

Once I was happy with the video, I downloaded the MP4 file and opened it in Adobe Photoshop. Due to the upscaling option on RunwayML, the video is twice the size of the source image, 2,816 × 1,536 pixels. I then split it into individual frames, made a copy of all the original frames, reversed the frame’s order, and combined them into the video to give it a continuous flow, forward then backward, when looped. You can also use FFmpeg for this task, which I detailed in the video and GitHub repository I referenced earlier. The added audio track is from Pixabay .

The final results were excellent, especially considering this all started with a significantly different source image.

Here is a second generated video; it was extended twice on Runway to 12 seconds. The fade to black at the end of the video and the audio track from Pixabay were both done post-production with Adobe Photoshop.

Here is a third generated video using a different source image. This video uses Runway RunwayML’s frame interpolation capability to generate smooth transitions between generated images (frames). The video is the result of creating frame interpolations between a series of similar generated images.

Creating Animated GIFs

MP4 format videos are easily uploaded to YouTube or shared on many popular social media sites. However, animated GIFs are smaller and quicker to upload to networks and can be set to loop indefinitely. Again, you can use Adobe Photoshop or GIMP to create animated GIFs of any size, compression ratio, and color palette.

The animated GIF used at the start of this post was created using Adobe Photoshop at reduced sizes of 512 pixels wide with increased compression to get under the 5MB file size limit of LinkedIn.

Conclusion

This post taught us to create short video animations from a single image. We started with licensed artwork from Shutterstock, created multiple variations of the image using Amazon Bedrock and Stability AI’s SDXL 1.0 image generation model, combined selected variations into a pleasing composition in Adobe Photoshop, and finally used RunwayML and PhotoShop to generate the short video.

This blog represents my viewpoints and not those of my employer, Amazon Web Services (AWS). All product names, images, logos, and brands are the property of their respective owners.