How to bring Painting to Life with AI

It is amazing what you can do with AI models these days with a gaming PC and NVIDIA card.

Goal

Wanted to take Raja Ravi Varma (The Milk Maid - 1904) painting and bring it to life with blink and smile.

Tools needed

1. Painting or Photo that you want to animate
2. StableDiffusion 2 with depth2img model.
3. EbSynth
4. Image editor (Used Inkscape)
5. Webcam (Logitech brio) and software to record short video of an actor in similar pose.
6. Background removal software if you don't have greenscreen. Used Erase.bg
7. Shot-cut video editor ( go from video to frames and back)
8. Gaming PC with Nvidia card (I had RTX 3090)

Steps

Original Painting-> Stablediffusion ImgtoImg using ( depth2image, face restoration)-> output image is saved as key frame.

Record video clip of actor in similar pose (costume doesn't need to be perfect).-> Use shot-cut to export video as individual png frames-> save to folder

Make sure your keyframe is perfectly aligned with one of the photos exported from shot-cut (Image editors to rescue. This took me the longest as I didn't see this coming!).

Use EbSynth -> provide keyframe folder and folder from shot-cut export->synth -> Magic happens and you get this.

Reference

https://huggingface.co/stabilityai/stable-diffusion-2-depth