Darth Vader Returns With the Help of AI, Meta Reality Labs Introduces QuestSim and NVIDIA’s CV-CUDA Accelerates Cloud Vision Apps

This week, AI is disrupting the entertainment industry with new advancements in video toonification, full body tracking with minimal sensors and saving the voices of iconic characters like Darth Vader. Meta Reality Labs introduced QuestSim, NVIDIA announced a toolkit for accelerating computer vision cloud apps and an AI-powered whale detection system hopes to eliminate ‘ocean roadkill’ altogether. Let’s dive in!

Research Highlights

• Researchers at Meta Reality Labs introduced QuestSim, a reinforcement learning framework that simulates full body motions for interactive and immersive experiences in AR/VR. Currently, real-time tracking of human body motion is limited by the sensor data captured by standalone wearable devices, but the QuestSim framework aims to prove that full-body simulation is possible even with sparse signals. The researchers demonstrated that by using high quality full body motion as dense supervision during training, a simple policy network can learn to output appropriate torques for the character to balance, walk, and jog, while closely following the input signals. VR enthusiasts and creators alike are excited about what this could mean for future gaming and animation production.
• Medical imaging researchers from NVIDIA won the HECKTOR22 challenge at MICCAI2022 for their proposed solution to head and neck (H&N) tumor segmentation. The task of the HECKTOR22 challenge was to accurately automate the segmentation of H&N tumors and lymph nodes from 3D PET/CTs pairs and the NVIDIA team’s solution achieved the best aggregated Dice accuracy out of all submitted projects. Recently, several radiomics studies proposed highly promising methods to better identify patients with an unfavorable prognosis in a non-invasive fashion via 3D PET/CTs imaging but were lacking further validation on larger cohorts. The challenge aims to expand on these studies to further the development of disease prognosis and treatment planning for H&N cancers.
• Researchers from Nanyang Technological University in Singapore introduced a novel image toonification framework called VToonify that has gained major interest from the computer graphics and vision community. Unlike other existing frameworks also built on StyleGAN, VToonify can synthesize high-resolution stylized videos from a low-resolution input and accepts non-aligned faces in videos of variable size as input. Currently, the ability to render high-quality “toonified” video remains too limited for full adoption by the entertainment industry but frameworks like VToonify have the potential to greatly alter the way your next favorite Pixar film might be made.

ML Engineering Highlights

• James Earl Jones, now 91, has signed over the rights to his Darth Vader voice to Respeecher, a Ukrainian AI company that trains text-to-speech ML models with the licensed and released recordings of actors who wish to move on from their iconic roles. Combining archival recordings and a proprietary AI algorithm, Respeecher most recently put together the Vader replacement voice for Disney’s Obi-Wan Kenobi and even Jones’ family was greatly pleased with the result of the returning galactic tyrant.?
• Whale Safe, an AI-powered whale detection system that alerts shipping companies to slow their boats in the presence of whales, is expanding its reach along the West Coast of the US. Funded by Marc Benioff, the system successfully dropped the number of whale deaths down to zero within its first year of operations last year in the Santa Barbara Channel. Researchers estimate that more than 80 endangered whales are killed by ships each year along the West Coast and conservationists are hoping tools like Whale Safe can help rewrite the doomed future for this important species.
• For the first time ever, the U.S. Open and other international tennis tournaments have replaced 200+ human judges with an optical tracking and camera collaboration technology called Hawk-Eye live. Initially introduced in 2020 as a measure to reduce personnel due to COVID-19, the technology has found a permanent home on the court due to its ability provide a far greater level of accuracy than its predecessors.

Open Source Highlights

• The world’s first AI-generated movie, Salt, is taking shape and viewers now have the ability to vote on what happens next in the multi-plot story. Everything from the graphics, voices and script are made using open-sourced AI tools like Stable Diffusion, Synthesia and GPT-3. Clips from Salt can be viewed on Twitter and offer an early example of how disruptive AI systems could be to moviemaking.
• Google AI introduced TensorStore, an open source C++ and Python library designed for reading and writing large multi-dimensional arrays. A core objective of TensorStore is to enable parallel processing of individual datasets while maintaining high performance. Multiple problems in scientific computing linked to the management and processing of enormous datasets in neuroscience have already been resolved using this library.
• NVIDIA introduced CV-CUDA, an open source toolkit for creating accelerated end-to-end computer vision and image processing pipelines “to process 10x the number of images at the same cost”. NVIDIA claims that CV-CUDA provides 50+ high-performance computer vision algorithms, a development framework that makes it simple to design unique kernels and zero-copy interfaces to eliminate bottlenecks in the AI pipeline.

Tutorial of the Week

• Hardware constraints getting in the way of training large models? Check out this tutorial on how to get unblocked by enabling native fully sharded data parallel in PyTorch!

Lightning AI Highlights

• Lightning CEO William Falcon was on last week’s Gradient Dissent, a podcast by Weights & Biases. Check out the episode, where he discusses machine learning frameworks, level setting expectations, and scaling PyTorch Lightning to Lightning AI.
• Our CTO Luca Antiga appeared on an episode of ‘How We Hatched’, from the Hatchpad Pair Program Podcast. Listen to him discuss a variety of topics, including what makes a successful member of the Lightning team!
• Did you catch members of the Lightning team at Free and Open Source Software (FOSS) United last week talking about the Lightning ecosystem? You can take a look at the resources they covered here.

Community Spotlight

Want your work featured? Contact us on Slack or email us at [email protected]

• Pietro Lesci’s Energizer is an Active-learning framework for PyTorch based on Lightning. With it, you can train any Lightning model using Active-learning with no code changes. It’s modular, easily extensible, provides a unified interface, and easily scales to multi-node/multi-gpu settings.
• O?uzcan Turan’s Lightning App is a demo of Satellighte, an image classification library used to classify satellite images. Because satellite data is highly variable, it is difficult to apply current object classification techniques to satellite datasets for tasks like land cover classification. The library seeks to address this challenge by establishing a light structure that nevertheless obtains robust results. This Lightning App bundles together a notebook, blog, and model demo where you can test out the model itself.
• A huge thank you to everyone in our community who has contributed to this issue. We’ve heard your feedback, and we’re in the process of removing APIs from the Lightning Framework that have already been deprecated in previous releases. Keep an eye out for future issues like this one — they’re a great way to get started with your first open-source contribution!

Don’t Miss the Submission Deadline

• ICLR 2023: 11th International Conference on Learning Representations. May 01-05, 2023 (Kigali, Rwanda).?Paper Submission Deadline:?Thu Sep 29 2022 04:59:59 GMT-0700.
• CVPR 2023:?The IEEE/CVF Conference on Computer Vision and Pattern Recognition. Jun 18-22, 2023. (Vancouver, Canada).?Paper Submission Deadline:?Fri Nov 11 2022 23:59:59 GMT-0800

Conferences

• ICIP 2022:?International Conference on Image Processing. International Conference on Image Processing. Oct 16-19, 2022 (Bordeaux, France)
• IROS 2022:?International Conference on Intelligent Robots and Systems. Oct 23-27, 2022 (Kyoto, Japan)
• NeurIPS?| 2022: Thirty-sixth Conference on Neural Information Processing Systems. Nov 28 - Dec 9. (New Orleans, Louisiana)
• PyTorch Conference: Brings together leading academics, researchers and developers from the Machine Learning community to learn more about software releases on PyTorch. Dec 2, 2022 (New Orleans, Louisiana)

Want to learn more from Lightning AI? “Subscribe” to make sure you don’t miss the latest flashes of inspiration, news, tutorials, educational courses, and other AI-driven resources from around the industry. Thanks for reading!