Create a AI model of your face and have fun with Stable Diffusion

Strike a pose for Van Gogh or Rembrandt !

How to create a custom AI model of yourself and use it in Stable Diffusion ?

Have you ever dreamed of posing for a Dutch genius, Monet, Picasso, Rembrandt, a famous Manga artist, Jim Lee or Greg Rutkowski maybe ? Well pinch yourself because your dream just came true.

To generate this image I trained a LoRa model of myself with 20 odd pictures and used it with Stable Diffusion’s 1.5 base model.

Stable Diffusion’s has an uncanny ability to recognize and apply the styles of artists of the past and the present whose work it was trained with. Painters, photographers, sculptors, you name it, there are hundreds of them.

Best thing is you can run and install Stable Diffusion on a regular M1 Macbook like mine and if you need speed you can make use of Google Colab to get the horsepower you need.

What a great time to be around !

Why I went with Stable Diffusion rather than MidJourney or Dall-E ?

Settling on a specific application, stack or library sometimes happens purely by chance and small things can make a difference.

I had heard of and tried out Dall-E and got fun and decent results but could never wrangle it to my desires. I had seen those amazing renders made with MidJourney and wondered if I could achieve something approaching this graphic awesomenes on my entry-level macbook M1 without having to pay a feeL

For all I know, when I applied for a slot in Mid Journey’s users pool, non-paying accounts were barred from prompting and it would not let me prompt. Plus MidJourney’s Chatbot in a discord channel was a bit perplexing. So that was a no-go to me so and I had to try other things.

This is when I stumbled upon StableDiffusion Online’s website which is a demo version of Stable Diffusion that runs on the browser and lets you try it out: https://stablediffusionweb.com/#demo

Upon further digging I saw you could even make SD run locally on a mac and this was the real deal for me. That had me in instantly and I don’t regret it.

Free to use

To give you some context the key factor that led me to using Stable Diffusion when I got started on this journey was that unlike MidJourney, while not properly open-source is free of charge.

Stable Diffusion actually has a CreativeML Open RAIL M License that limits certain use cases but it’s code is public unlike that of OpenAI’s Dall-E.

To me this meant no need to sign up for any plan to start creating. Plus everything you create is yours, you don’t have to share it to the world until you deem it worthy of public attention.

Training a AI model of yourself in a blink

I encourage you to follow along Laura Carnevali’s series of tutorials on model training, she has clear explanations and to-the-point no-nonsense approach.

There are various methods to install and run SD with different tradeoffs. Web UI, Comfy UI, DreamBooth, in Colab notebooks or on your machine. Laura has tuts about all these and she has you covered whether you use a mac or PC.

For that matter, AI Image generation is a fresh but trending topic with a lot of content producers, a growing community of people who want to leverage these tools and make cool stuff. There are countless tuts around about LoRa, SD, extensions, so pick yours and debug as you go )

I suggest you really try out Google Colab to run SD in the cloud. It is way way quicker (we are talking minutes instead of hours and seconds instead of minutes) than running it locally I guess even if you already have a GPU. So much so that it might be worthwhile to invest a few bucks to get computing power to match your project’s needs.

.safetensors or .ckpt : which to chose ?

To get a perfect shot of yourself in a Monet painting amidst a landscape of water-lilies or in a H.R. Giger alien spaceship, you can create your own checkpoint model and merge it with a base model like version 1.5. But there is a better, lighter solution.

Instead of merging full-fledged checkpoint files and getting a 2GB file like I did when I knew no better, you can actually achieve the same result with a 20MB file called a LoRa. Call that a diet !

A LoRa is a Low-Rank adaptation of a larger model. They can be used to portray celebrities or create a background in a specific style. Either way you have it, checkpoint or Lora?this should work.

Say cheese ! ????????

LoRa and checkpoint files can be trained with a surprisingly low number of pictures. Some 20 images can give good results, maybe even as low as 10. But before you upload your pictures I advise you to pay attention to the quality of the input you provide as you know the saying: “garbage in, garbage out”.

Try different expressions, distance from camera,?light settings and so forth. Give it your best shot ! Strike some poses and when you have 20 beautiful pictures of yourself hit the launch button. Here you can grab a cup of coffee. And prepare to wait. And wait. And wait. Hold on … No, keep on waiting some more. Eventually you’ll be able to marvel at an AI rendition of yourself that is you without being you. The training can take up anywhere from 45mn to 1 hour so be patient, you’ll be rewarded.

If the results are not up to your expectations, try a combination of different learning-rate and steps until you get a picture-perfect render.

A few moments later ..

???While training is ongoing, try to keep the web UI page alive or it might disconnect and you’d have to start it all back again.

How to use your custom LoRa in Web UI

In Web UI, toggle down the “display additional addons” tab and click on a lora’s name. This will append it to the positive prompt in the form of: lora:your_loras_name:1

When generating images try tweaking the CFG rate. It lets SD be more or less creative. Pick a different sampler. Find the sweet spot when you get a realistic rendering that honors your inner and outer beauty !

From then on you’re in uncharted territory and your imagination is the only limit. Time for you to create amazing stuff ! ???????????????

SD running too slow ? Run it on a powerful GPU with Google Colab !

In case it’s still going too slow for your liking, and you’re not gonna wait minutes to render an image like it’s 1995 and you’re downloading movies overnight. No bro, it’s 2023 and Google Colab can provide more than a fix but comes at a (still affordable) price.

Colab’s notebooks let you run the show straight from the cloud and make use of Nvidia’s top of the shelf GPU capacities to make you.

They have a $10 monthly plan which I opted for. There are a few options to try out and also an option to pay for the thing as you use it.

Be careful to close out your session when you’re finished.

???The Colab monthly plan gives you 100 computing units. I have to warn you that they may go down quickly if you have several sessions running concurrently.

If this happens, have a look at the “See resources” tap at the top right near the RAM level indicator.

You can use Colab to train models and LoRa, that will be much faster than using your mac’s CPU. Think about it !

Use this notebook to train your LoRa : https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb

???Useful links :

*> Laura has a great youtube channel with tutorials on AI Image generations

Install Stable Diffusion locally on macbook with Automatic1111* Spreadsheet with names of artists recognized in SD v1.5. (Awesome) Get better at prompting

That’s all folks !

Give me a ?? if this was useful.

Comment, like, share, and have fun !