Using Stable Diffusion + ControlNet for creative variations from a photo shoot

The more we collectively use and explore the space AI image generation space the more control we want from it. Yes, we can create beautiful images using text prompts, selectively regenerate parts of it, use in-painting to expand on images. But what if we already have a solid vision of what we want and we want to scale that vision into the infinite possibilities without losing what made the vision grand?

Well we now have a answer to that with Stable Diffusion + ControlNet . To put it simply ControlNet works with Stable Diffusion to further fine tune and guide the output of Stable Diffusion. It allows for finer spatial consistency to tell the Stable Diffusion model what to do especially when you already have a strong vision of how the image should be composed.

The amazing thing about ControlNet is that is already has a few pre-trained models that control the image-to-image generated based on different parameters. For example edge detection, depth maps, sketch processing or even human poses. There is a bit of exploration needed to match the correct control model to the image you are using. Most of that is discussed in the original paper

This is a short post but I think you are probably hungry to try it out especially if you are already setup with Stable Diffusion but if not then here are a few links on getting it set up.

I recommend setting up Stable Diffusion with the Automatic111 Web UI as it is the most widely supported UI and makes interfacing with Stable Diffusion easier. Warning you do need a bit of a stomach for installing all kinds of tools to your Windows PC: https://rentry.org/voldy

You can then use this WebUI ControlNet integration to install ControlNet as a extension in Web UI: https://github.com/Mikubill/sd-webui-controlnet

Do note that you will need atleast a Nvidia Ampere based card with 4gb of VRam to run this.

#stablediffusion #controlnet #generativeai #image #photoshoot #variations #artisticvariations #prototyping