Meta's GenAI Developments - A Closer Look

?? Launch of Safety-Tuned Chameleon Models for Mixed-Modal Input - Meta has recently released the Chameleon 7B and 34B safety-tuned models that is designed to handle mixed-modal inputs and produce text-only outputs. These models has a unified architecture that allows seamless processing of text and images, a move from traditional models that use separate encoders or decoders for each modality. The Chameleon models are part of Meta's efforts to democratize access to advanced AI technology, and they are released under a research-only license to ensure responsible use. The Chameleon models have been safety-tuned to minimize harmful outputs and reduce biases. This makes them suitable for various applications, such as customer support, content moderation, and educational tools. Meta has emphasized the importance of inclusive access to these technologies, aiming to spur innovation and development in the AI community.

?? Meta Launches JASCO, AN Advanced AI for Text-to-Music Generation - JASCO (Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation) is Meta's innovative GenAI model that is designed to enhance music creation by accepting various inputs such as chords and beats. This multi-input conditioning allows for precise control over different aspects of the music, such as melody, harmony, and rhythm, making it a versatile tool for musicians and producers. JASCO's performance is on par with other state-of-the-art text-to-music models, offering high-quality output with significantly better control. Meta plans to release the pretrained JASCO model under a non-commercial Creative Commons license, encouraging researchers and developers to explore and build upon this technology. This initiative aligns with Meta's commitment to advancing open science and responsible AI development.

?? Meta Llama 3 Hackathon with Cerebral Valley - The first-ever Meta Llama 3 Hackathon was recently hosted together with Cerebral Valley, marking a significant event in the tech community. The hackathon garnered substantial interest, with over 1,200 applications received. Out of these, 350+ attendees were selected to participate. During the intense 24-hour event, participants showcased their skills and creativity, building a range of impressive projects. This hackathon highlighted the innovative potential of Meta Llama 3 and brought together a vibrant community of developers and tech enthusiasts.

?? Launch of AudioSeal - Meta has introduced AudioSeal which is an advanced audio watermarking model designed for the localized detection of AI-generated speech thus marking a significant leap in audio authenticity verification. AudioSeal has a robust and rapid detection capabilities, outperforming previous methods like WavMark by up to 485 times in speed and offering enhanced resistance to audio manipulations. It features a novel perpetual loss inspired by auditory masking, ensuring the watermark is imperceptible while maintaining high detection accuracy. AudioSeal supports multi-bit watermarking, allowing precise attribution of audio to specific models or versions without impacting detection. This technology is crucial for industries requiring high audio integrity, such as media, security, and communications.

?? Multi-Token Prediction Models for Code Completion - Meta has recently launched its pre-trained Multi-Token Prediction (MTP) models for code completion that can predict multiple tokens simultaneously thus enhancing performance and reducing training times. These models were released under a non-commercial research license on Hugging Face, significantly outperform traditional single-token models by 17% on MBPP (Python coding tasks) and 12% on HumanEval benchmarks thus generating output three times faster. The MTP approach addresses the limitations of the teacher-forcing method by better capturing long-term dependencies in code, potentially improving tasks beyond code generation such as creative writing. While democratizing powerful AI tools Meta also acknowledges the risks of misuse, emphasizing their research-only nature to mitigate potential threats.

?? Meta Unveils MAGNeT, A Revolutionary High-Speed Text-to-Audio Generation Model - MAGNeT is developed by researchers at Meta is a new non-autoregressive transformer model designed for text-to-music and text-to-sound generation that is capable of producing high-quality audio at seven times the speed of state-of-the-art models. Unlike the traditional non-autoregressive models, MAGNeT does not require semantic token conditioning, model cascading, or audio prompting, utilizing a single transformer for full text-to-audio conversion. The training and inference code for MAGNeT has been released as open source, and this groundbreaking work was presented at the International Conference on Learning Representations (ICLR) 2024 in May, marking a significant advancement in the field of audio generation.

?? Niantic’s Peridot, How Meta Llama Models Are Changing the Way We Play with Virtual Pets - Niantic Inc. is leveraging Meta's Llama 2 models to enhance the interactive experience of their AR game, Peridot. This innovative use of GenAI allows the game's virtual pets, known as "Dots," to exhibit lifelike, unpredictable behaviors that mimic real animals. The integration of Llama 2 enables each Dot to have unique characteristics and react dynamically to its environment, creating a more immersive and engaging experience for players. The Niantic Lightship ARDK further enhances this experience by allowing Dots to recognize and interact with real-world objects like flowers, food, and pets. This combination of AR and AI helps the Dots to display a wide range of emotions and behaviors, from curiosity and joy to mischief, making them feel like real companions. Niantic plans to continue expanding the capabilities of Peridot by exploring new GenAI applications, aiming to elevate player interactions across devices. This approach not only pushes the boundaries of AR gaming but also opens up new possibilities for the gaming industry to leverage AI in creating more complex and engaging virtual experiences.

?? Meta's Llama Enhances FoondaMate for Better Student Support - Meta AI has enhanced FoondaMate which is an AI study buddy with its advanced conversational model Llama. This integration enables students to get detailed academic support via WhatsApp and Messenger, making learning more accessible and interactive. The upgrade determines AI's expanding role in education, offering personalized help for complex subjects and assignments through widely used messaging platforms.

?? CyberSecEval 2 Cybersecurity Evaluation Suite for LLMs - CyberSecEval 2 was released by Meta AI on April 19, 2024. This comprehensive benchmark suite is designed to assess the security risks and capabilities of LLMs. It includes new testing areas such as prompt injection and code interpreter abuse. Meta AI's researchers introduced the safety-utility tradeoff that is quantified by the False Refusal Rate (FRR), to evaluate how often LLMs mistakenly reject benign prompts while effectively refusing unsafe ones. This benchmark helps ensure the robustness of LLMs by categorizing tests into logic-violating and security-violating prompt injections, vulnerability exploitation, and interpreter abuse evaluations.

?? FAIR Releases Comprehensive Guide on Vision-Language Models (VLMs) - FAIR (Facebook AI Research) has released a comprehensive guide on Vision-Language Models (VLMs) titled "An Introduction to Vision-Language Modeling." This guide provides detailed insights into the workings of VLMs, how to train them, and the various approaches to evaluating these models. While the primary focus is on mapping images to language, the guide also explores extending VLMs to handle video data. The guide explains the basic principles of VLMs, highlighting how these models integrate visual and linguistic data. It covers the methodologies for training VLMs, including the use of large datasets, pretraining, and fine-tuning. Various metrics and methods for evaluating the performance of VLMs are discussed, ensuring a robust understanding of their capabilities and limitations. The guide delves into the complexities of extending VLMs to videos, addressing challenges such as temporal dynamics and maintaining contextual relevance. FAIR has released this guide in collaboration with several partners, aiming to foster a deeper understanding of the mechanics behind mapping vision to language. This collaborative effort is intended to push the boundaries of research and application in the field of Vision-Language Modeling.

?? Startup Accelerator Launch in Europe - Meta in partnership with Hugging Face and Scaleway has announced a new accelerator program for AI startups in Europe, based at STATION F in Paris. The initiative aims to boost AI innovation and business growth while strengthening the European tech ecosystem. Selected startups will benefit from technical mentoring by Meta FAIR's research teams thus having access to Hugging Face’s platform and tools, and Scaleway’s computing power to support their open-source AI projects. The applications are open until August 16, 2024, for the interested startups to apply.

?? PlatoNeRF Advanced 3D Reconstruction Technology – The development in 3D reconstruction technology was presented by researchers from MIT and Meta at the Conference on Computer Vision and Pattern Recognition (CVPR) 2024. PlatoNeRF utilizes a combination of neural radiance fields (NeRF) and single-photon lidar to achieve accurate 3D reconstructions from a single viewpoint. This method can discern both visible and occluded geometry without relying on data priors or controlled lighting conditions, making it highly suitable for use in consumer devices such as smartphones and AR/VR headsets. The approach leverages two-bounce lidar, which captures the time it takes for light to bounce twice in a scene before returning to the sensor. This method provides detailed information about both the visible parts of a scene and the occluded regions by analyzing shadows and light reflections. The integration of multibounce lidar with machine learning enables PlatoNeRF to outperform traditional methods that either rely solely on lidar or on neural networks with RGB images, especially in low-resolution scenarios typical of consumer-grade sensors. This advancement represents a significant step forward in the field of 3D scene understanding and has potential applications in various industries including autonomous vehicles, AR and robotics industry.

?? Introducing Meta 3D Gen - Meta has introduced Meta 3D Gen which is a groundbreaking text-to-3D research project that enables the generation of 3D models with high-quality geometry and textures from text descriptions. This innovative technology significantly advances the field of 3D asset creation by offering Text-to-Mesh Generation. Meta 3D Gen can produce high-quality 3D assets, complete with detailed geometry, high-resolution textures, and PBR (Physically Based Rendering) materials. This end-to-end generation surpasses previous state-of-the-art solutions in both quality and speed, achieving results 3-10 times faster. Meta's research has two key components.

o?? Meta 3D AssetGen that is focused on generating 3D models from textual descriptions ensuring the creation of detailed and accurate 3D geometries.

o?? Meta 3D TextureGen specializes in generating high-quality textures and retexturing artist-created or generated assets with AI assistance thus enhancing the visual fidelity and realism of the 3D models.

?? Launch of Llama Impact Innovation Awards - Meta has launched the Llama Impact Innovation Awards to recognize and support organizations using Meta Llama models for social impact in Africa, the Middle East, Turkey, Asia Pacific, and Latin America. The program offers awards up to $35,000 USD to organizations tackling pressing regional challenges. The applications for this are open until July 26, 2024.

?? Nymeria, Largest Multimodal Egocentric Motion Dataset in Natural Environments - The Nymeria dataset that was recently introduced is one of the most extensive and diverse collections of multimodal egocentric data on human motion captured in natural environments. This dataset includes 300 hours of daily activities recorded from 264 participants across 50 locations, utilizing devices such as Project Aria headsets, miniAria wristbands, and XSens mocap suits. These devices provide synchronized and localized 3D motion data, along with RGB, grayscale, eye-tracking cameras, IMUs, magnetometers, barometers, and microphones. Nymeria aims to advance research in areas like egocentric body tracking, motion synthesis, and action recognition. It offers a comprehensive set of 1200 recordings, world-aligned 6DoF transformations for all sensors, and 3D scene point clouds with calibrated gaze estimation. It also includes 310.5K sentences in 8.64M words for motion-language descriptions, enhancing the dataset's value for language-related tasks.

?? Relightable Gaussian Codec Avatars - The Relightable Gaussian Codec Avatars that was recently presented by Meta represents a significant advancement in creating high-fidelity, relightable head avatars. These avatars can be animated with novel expressions and adapted to various lighting conditions in real-time. This is achieved through a combination of 3D Gaussian geometry and a novel learnable radiance transfer model, allowing for the capture of intricate details like hair strands and skin pores. One of the notable breakthroughs is the use of Gaussian splatting, which enables efficient and realistic rendering of these avatars under changing lighting conditions. This method not only improves the fidelity of the avatars but also maintains real-time performance, making it suitable for applications in VR and interactive environments. The avatars were showcased at CVPR 2024, highlighting their ability to handle diverse materials and complex lighting scenarios. This development is part of Meta's broader effort to enhance telepresence technologies, potentially enabling users to create photorealistic avatars using just a few photos taken with a smartphone, instead of requiring extensive 3D scans in specialized studios.

?? Meta’s Llama 3 Powered by NVIDIA is Transforming Healthcare & Life Sciences - Meta’s Llama 3 optimized by NVIDIA is revolutionizing healthcare and life sciences. The NVIDIA NIM (Neural Inference Microservice) version of Llama 3 is being adopted by over 40 healthcare companies for applications like surgical planning, drug discovery, and clinical trials. Some of the key users include Deloitte for drug research, Transcripta Bio for drug discovery, and Activ Surgical for real-time surgical guidance. These applications enhance digital biology, clinical trials, digital surgery, and digital health by improving efficiency, patient outcomes, and innovation.

?? URHand, Universal Relightable Hands - The URHand Universal Relightable Hands project was presented at CVPR 2024 introduced a groundbreaking model for creating highly realistic, relightable hand models. This model was developed by researchers from Meta and Nanyang Technological University can adapt to various viewpoints, poses, identities, and lighting conditions, and it allows for quick personalization from simple phone scans. The innovation lies in its hybrid approach, combining a physics-based branch for geometry refinement with a neural branch that utilizes a linear lighting model, making it possible to render realistic hand images under diverse lighting conditions. It utilizes 150 cameras and 350 LED lights to capture dynamic hand movements, ensuring detailed and accurate hand models. The physical branch focuses on refining geometry and shading features, while the neural branch predicts final appearance using a spatially varying linear lighting model. It enables scalable cross-identity training, maintaining high fidelity and sharp details, outperforming previous methods such as Relightable Hands and Handy. It allows for rapid creation of personalized hand models from phone scans, making it accessible and user-friendly for various applications. URHand's ability to generalize across different conditions and its high-quality rendering capabilities make it a significant advancement in the field of digital hand modeling and rendering.

?? RoHM-Robust Human Motion Reconstruction via Diffusion - RoHM introduces a new method for reconstructing 3D human motion from single RGB or RGB-D videos. It uses diffusion models to tackle noise and occlusions, improving motion reconstruction accuracy. This method separates the task into handling global trajectories and local motions, employing an iterative denoising approach. RoHM has shown superior performance in motion reconstruction, denoising, and infilling on various datasets and is faster during testing compared to existing methods.

?? HybridNeRF, An Efficient Neural Rendering via Adaptive Volumetric Surfaces - HybridNeRF is a method designed to enhance neural rendering by combining surface and volumetric representations. By using surfaces for most objects and volumetric methods for challenging regions, HybridNeRF achieves efficient and high-quality rendering. This approach improves error rates by 15-30% and supports real-time framerates of at least 36 FPS at 2Kx2K resolution, outperforming existing methods in both speed and quality. The technique is validated on datasets such as Eyeful Tower and ScanNet++, demonstrating superior performance in rendering complex scenes, especially in handling specular effects and reflections.