Did Meta Researchers Just Prove That RLHF Is Not Needed?

Meta researchers trained an LLM without reinforcement learning or human preference modeling. NVIDIA partners with Microsoft to accelerate AI efforts, The White House takes more steps to advance responsible AI and we’re teaching you how fine-tune LLMs on a custom dataset. Let’s dive in!

Research Highlights:

• Meta researchers conducted a study on LLMs and their training process. The researchers trained a 65B parameter LLaMa language model called LIMA using only 1,000 curated prompts and responses without reinforcement learning or human preference modeling. LIMA is claimed to have achieved impressive performance, showing the ability to follow specific response formats and generalize well to unseen tasks, indicating that most knowledge in large language models is learned during pretraining, with limited instruction tuning data needed for high-quality output. In a controlled human study, LIMA's responses were claimed either equivalent or preferred over GPT-4 in 43% of cases, with even higher rates of preference when compared to Bard and DaVinci003.
• University of Washington researchers introduced QLoRA, a finetuning approach that is claimed to significantly reduce memory usage while maintaining full 16-bit finetuning task performance on a single 48GB GPU. Their model family, named Guanaco, is reported to surpass previously released models on the Vicuna benchmark, achieving 99.3% of ChatGPT's performance with just 24 hours of finetuning on a single GPU. QLoRA’s authors also claims that it incorporates techniques such as a new data type, double quantization, and paged optimizers to save memory without compromising performance, enabling the finetuning of large-scale models.
• Stanford researchers introduced Sophia, a new second-order optimization algorithm called Second-order Clipped Stochastic Optimization, aimed at reducing the time and cost associated with language model pre-training. Sophia utilizes a lightweight estimate of the diagonal Hessian as a pre-conditioner, promising to enable efficient updates and element-wise clipping to control the worst-case update size. By estimating the diagonal Hessian every few iterations, Sophia is claimed to achieve a 2x speed-up compared to the widely used Adam optimizer in language modeling tasks, without depending on the condition number of the loss.

ML Engineering Highlights:

• NVIDIA partners with Microsoft to integrate AI software into Azure ML, bringing deep learning to Windows 11 PCs. This collaboration is aimed to accelerate AI efforts, simplify development and deployment, and offer access to NVIDIA's AI enterprise and Omniverse Cloud on the Azure marketplace. GPU-accelerated deep learning frameworks will be enabled on Windows 11 through WSL, providing a local AI development option alongside large-scale training on Azure.
• The US Presidential Administration is taking new steps to advance responsible AI that prioritizes individuals' rights, safety, and the public good. This includes an updated roadmap for federal investments in AI research and development, a request for public input on AI risks and opportunities, and a report on AI in education. The Administration aims to promote responsible AI innovation, address risks, and ensure U.S. leadership in trustworthy AI systems, while engaging with workers and the public on critical AI issues.
• Apex.AI, a middleware company, announced that its operating system will be used in autonomous vans developed by Volkswagen Group's commercial vehicles division for MOIA's ridepooling service. The middleware system acts as the vehicle's operating system, similar to iOS or Android on a smartphone, and enables the autonomous functionality of the vehicles. MOIA aims to launch an autonomous ride-hailing service using Apex.AI's software in the coming years.

Open Source Highlights

• Meta introduced its Massively Multilingual Speech (MMS) project, an AI language model that can recognize over 4,000 spoken languages and produce speech in over 1,100 languages. The project aims to preserve language diversity and is open-sourced to encourage further research and development by the AI community. Meta used unconventional data sources, such as audio recordings of religious texts, to train the model, and the results show that MMS outperforms existing models in terms of word error rate and language coverage.
• Ledger, a hardware wallet provider, has addressed concerns raised by the community regarding its recently announced Ledger Recover service. In a Twitter space discussion, the Ledger team emphasized their commitment to security and discussed plans to improve transparency, including open-sourcing more of their code. Ledger's Chief Technology Officer, Charles Guillemet, outlined the company's open-source roadmap, which involves making the Ledger Recover protocol's white paper and firmware open source for review and verification by experts and developers. As a result, Ledger has paused the release of their recovery service to prioritize transparency and address community concerns.
• Microsoft is expanding its ecosystem of copilot applications, which are intelligent assistants powered by AI and large language models. The copilot applications are aimed to assist users with complex cognitive tasks, such as writing code, generating images, or planning events. Microsoft is introducing new copilot experiences across its core products and services, and developers will have the ability to build their own copilots using the company's AI development framework and plugin capabilities. The goal is to make copilots a standard expectation for how software works in the future.

Tutorial of the Week

Learn how to fine-tune LLMs on a custom dataset! In this tutorial we used Lit-Parrot, a nanoGPT based implementation of the GPT-NeoX model that supports –StableLM, Pythia, and RedPajama-INCITE model weights.

Don’t Miss the Submission Deadline

• ICCVS 2023: The 14th International Conference on Computer Vision Systems. Sep 27 - 29, 2023. (Vienna, Austria). Submission Deadline: Mon May 29 2023
• AI World Barcelona 2023: International Conference?dedicated to the field of?generative?AI?and?autonomous?agents. September 7 - 8, 2023. (Barcelona, Spain). Submission Deadline: Wed Jun 07 2023 16:59:59 GMT-0700
• CoRL 2023: International conference focusing on the intersection of robotics and machine learning. Nov 6 - 9, 2023. (Atlanta, Georgia). Submission Deadline: Fri Jun 09 2023 04:59:00 GMT-0700
• ACML 2023: The 15th?Asian Conference on Machine Learning. Nov 11 - 14, 2023. (Istanbul, Turkey). Submission Deadline: Sat Jun 24 2023 04:59:00 GMT-0700
• ICMLA 2023L: The 22nd International Conference on Machine Learning and Applications. Dec 15 - 17, 2023. (Jacksonville, Florida). Submission Deadline: Sat Jul 15 2023

Want to learn more from Lightning AI? “Subscribe” to make sure you don’t miss the latest flashes of inspiration, news, tutorials, educational courses, and other AI-driven resources from around the industry. Thanks for reading!