Puppeteering digital characters in real time

As a technical artist with a career in animation I’ve built character rigs and control systems that check all the boxes for the animators I worked with. However, taking risks or testing new paradigms is often not an option during production due to the time constraints. Which leads to repeating the same tricks over and over again.

However, tinkering with new ideas in a low risk environment… that’s where the fun lives!

One of my passions is exploring different ways to drive digital characters, particularly in real time.

For my last sketch I’ve connected an iPhone with Blender to drive a Grease Pencil face drawing. I’m currently writing a document explaining how to set it up, including the use of add-ons and code. I'm also preparing a Blender scene with more detail than what I published a few weeks ago. I will soon share both so everyone that is interested can experiment with the technique.

As a prologue to that document, here’s a list of experiments I’ve done with alternative character control methods, which hopefully can be inspiring to you.

IPAD + 3DSMAX

In 2011, I connected a GUI I made in my iPad 2 with a character I built in 3DSMAX, who I made for an online animation school. I used TouchOSC to design the iPad interface and rtpmidi to pass the data from the iPad to my Windows machine. And even though it didn't make it to the final prototype, I also explored pureData to create some “advanced” behaviours, like resetting all the sliders when pressing a button on the iPad interface.?

Inside 3DSMAX, each face expression/blendshape has a Motion Capture controller, which enabled me to control a face performance in real time, and even recording it.

You might think there really existed a straight path to do all this, but there was not. These results come after hours of fiddling, testing, getting frustrated and trying other ideas. When exploring "uncharted" territory always remember to have fun during the process!

The character in itself looks pretty basic, but it was the first time I designed, modeled and rigged a character from scratch for a client. I didn’t have a lot of experience with facial blendshapes back then, so I was very conservative about their range. I was more worried about all the expressions mixing properly than about having a character with appeal or a decent motion range. However, the experiment got some traction online and it was even featured in the magazine 3DWorld in September 2012.

UNITY + RADIO CONTROLLER

In 2018, while working at Disney Animation, I wanted to learn a little bit of Unity on my own time. I just also delivered a character rig for a client on the side, so I thought it would be cool using a USB game controller to play with its face. I didn’t know Unity, I didn’t know C# and I didn’t know if this was even possible!?

In my opinion, one of the best ways to learn is by setting a defined goal that forces you to scavenge and look online for different steps to keep advancing.

Whether it’s written tutorials, videos with instructions, stale forum posts, software documentation or even conversational AIs, I don’t think there’s any “wrong” information to be found. Because even when that path sends you to a place with no relevant data about what you are trying to do, it could open tangents where you discover techniques, works and knowledge that you were not aware of. This “I know nothing, let’s do it!” is by far one of the challenges I enjoy the most, more than the result itself.

While the code I wrote is a little bit embarrassing, I could figure out how to get closer and closer to my goal. I even wondered if I could use the orientation of my Android phone to drive the head rotation. And that’s why you see my phone with Unity Remote strapped with a rubber band to the radio controller.

I used the radio controller instead of a regular USB gamepad (which can be used to achieve the same results) because I saw artists driving animatronics with them, and I love animatronics! I bought a second hand controller just for this test, so having it laying around of course I had to get into FPV quadcopters next.

What I'm doing in the video below is controlling the jaw bone with a joystick and a few blendshapes with another joystick and some knobs. The phone sends accelerometer data to the host computer, which is used to drive the head rotation.

After seeing these results, the school I mentioned before hired me to deploy a similar system to one of their characters, to show it off in one of their outreach events. I wish I was there to see it, but I was told the attendees really enjoyed playing with it!

In this instructional video (in Spanish) I explain how to puppeteer the character to the team.

UNITY + IPHONE

In 2019 I started working at Genies, where I got to work on developing a stylized avatar system. In 2020, for my Christmas break, I decided I wanted to create a standalone iOS app in Unity to driving one of our character archetypes with ARKit. Again, just for fun.

I learnt the steps that were involved in taking a “published” Unity project through XCode to move it to my iPhone, using my wife’s iMac for the process. And I wanted to add a twist to the project: at the time driving a blendshape-based face rig with ARKit signals wasn’t a novelty, but I hadn’t stumbled upon a bone-based face rig (which is what we were using at the time) being driven with said signals.?

Using Unity’s Animation Layers, a couple of hidden properties and C# code I could slap together something solid enough to record a few clips to show off the system.

These are just a few examples of how I've explored alternative ways to bring digital characters to life. But the journey doesn't stop here! I'm excited to share the details of my latest experiment with you soon, complete with a Blender scene and instructions so you can try it out for yourself.

With the rise of VTubers and their specialized software, today it is even easier to get some of these ideas up and running. So dive in, have fun, and don't be afraid to share your own creations with the world!

I have a few more things cooking, so stay tuned for more updates!