Provide control is the keyword on AI race

There are multiple research approaches regarding the future of AI, but what is the real challenge??

Lots of companies provide the users the possibility to create photorealistic images, even amazing videos. That makes everyone question the future of the film industry.

Generative AI creates photorealistic images using an extraordinary amount of computational resources, yet that won't be a problem in the near future because companies like Nvidia have been working to enhance the power of AI processing with mixing precision training like the fp16 approach. Sooner than later we will be able to run complex AI models locally.

Nevertheless the professional film industry requires a lot of control of the process, story telling depends on that. Artists use a lot of tools to control the CG process, directors use a huge production infrastructure to direct a film.??

Real control comes from the sophistication of tools. Because, fires were not useful until someone created a fire pit.?

The question is, who is going to provide that level of control of AI tools to the film industry?

ComifyUI LivePortrait?

ComyfyU is a powerful node interface that allows users to design and execute advanced stable diffusion pipelines without needing to code, based on an intuitive graph system. ComfyUI supports various versions of Stable Diffusion. It can be configured to work on GPUs or CPU-only, and enables loading and saving of models, embeddings, and previous workflows.

Recently a contributor development release ComyfyUI LivePortrait, it is a set of nodes designed for ComfyUI that allows users to create animated portraits. This tool uses various machine learning models to generate realistic animations of facial features such as eyes and lips, providing a dynamic and lifelike appearance to still images. The project includes components for feature extraction, motion extraction, warping, and stitching, which work together to animate the portraits.

GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians

The core idea is a dynamic 3D representation based on 3D Gaussian splats that are rigged to a parametric morphable face model. This combination facilitates photorealistic rendering while allowing for precise animation control via the underlying parametric model.

This is what will really provide control to artists, a way to direct a digital character in a specific way

The input method is a multiview video recording of a human head, processed using a photometric head tracker to fit flame at each step , they established a local coordinate system for each flame triangle and initialize a 3D gaussian splat at the origin this enables the gaussian splats to move with the triangles when the flame mesh is animated, they render the gaussian splats an use the color loss to optimize their color? and opacity, then they optimize each gaussian flats local scaling, position and? rotation to obtain a more accurate geometric representation. They introduce a binding inheritance strategy that allows to densify and prune gaussian splats ensuring the highest fidelity without losing control ability.

Vid2Avatar: 3D Avatar Reconstruction from Videos via Self-supersived Scene?

This is another interesting paper that proposes a new dataset called SynWild to evaluate the monocular human surface reconstruction task. Dynamic human subjects are captured in a dense multi-view system and reconstructed with detailed surface geometry to achieve robust and detailed 3D reconstruction of the human even under challenging poses and environments without requiring external segmentation methods.

Finally, generative AI models has a lot of problems using 3D spatial references, that's the reason it still be complicate to turn around a character head in AI face wrapping, but it is about of time to link 3D ML models, gaussian splating and node interfaces to create a new era of Ai tools. That will be the real revolution, a new generation of artists with sophisticated tools to make what they want instead of what AI wants.?

Most experienced artists have these challenges. They will have to make a decision to learn a different way of thinking about the tools. I have been talking with different artists around the film industry and there is a lot of anxiety. Nobody really understands where the future is going, including me, but I am pretty sure that AI tools will be a part of that future.. New top artists will need a basic understanding of ML, a high level of adaptation and a huge amount of artistic background to create amazing artcraft. Because art is not a tool, art is a way to see the world we live in.?