Introducing GLM a General Language Model

Reference Articles: https://arxiv.org/abs/2103.10360 , https://arxiv.org/abs/2210.02414

Introduction

The world of artificial intelligence and natural language processing has witnessed a remarkable evolution with the advent of Large Language Models (LLMs). Among the front-runners in this revolution is GLM-130B, a bilingual pre-trained language model, making waves in the AI community.

Published as a conference paper at ICLR 2023, this research not only highlights the technical superiority of GLM-130B but also delves into its ethical and practical implications in the realm of AI and language processing.

Technical Overview

Performance Benchmarking

GLM-130B sets a new standard in language model performance. In a comparative study with the 260B ERNIE Titan 3.0, the largest existing Chinese monolingual language model, GLM-130B consistently outshines across 12 different tasks.

This is particularly evident in its handling of two abstractive Machine Reading Comprehension (MRC) datasets, DRCD and CMRC2018, where it surpasses ERNIE by at least 260%. Such a leap in performance is attributed to GLM-130B’s pre-training objectives, which seem to align more naturally with the demands of abstractive MRC.

Pre-Training Insights

The pre-training phase of GLM-130B adheres to the transformative scaling laws observed in transformer-based language models. Starting from smaller models like the 1.5B GPT, scaling up to the 100B-scale GPT-3, a pattern emerges where new capabilities arise as the model size increases.

However, GLM-130B stands out by breaking the norm of limited accessibility that plagues many 100B-scale LLMs. It champions the cause of high-quality, open-sourced LLMs, aiming to make advanced AI tools more accessible to the broader research community.

Transferring Knowledge

In the context of transferring knowledge, GLM-130B moves away from traditional fine-tuning methods. Due to the massive size of LLMs, the focus has shifted towards prompting and in-context learning.

Although parameter-efficient learning and prompt tuning are emerging trends, their exhaustive exploration in relation to GLM-130B is slated for future research.

Efficient and Fast Inference

GLM-130B also emphasizes efficient and fast inference, a critical aspect for the practical deployment of LLMs. It explores avenues like distillation, quantization, and pruning, with a notable achievement in INT4 weight quantization.

This advancement enables GLM-130B to operate on commonly available hardware, such as 4×RTX 3090 or 8×RTX 2080 Ti GPUs, making it more accessible for practical use and research.

Technical Deep Dive into GLM-130B's Advanced Architecture and Innovations

Pipeline Parallelism and Model Efficiency

GLM-130B showcases an advanced implementation of pipeline parallelism, significantly reducing inefficiencies seen in traditional models. The research contrasts three pipeline strategies: Naive Pipeline, GPipe, and Pipedream implementations, with GLM-130B utilizing the latter.

The Pipedream approach minimizes pipeline 'bubble time', allowing for overlapping computation and communication, thus enhancing training efficiency. This is critical in harnessing the full potential of hardware, particularly in balancing between pipeline model parallelism and tensor model parallelism.

Inference Acceleration

To optimize inference speed, GLM-130B was implemented in C++ based on NVIDIA's FasterTransformer.

This involves optimizing time-consuming operations, reducing GPU kernel calls, and using specialized algorithms for improved performance. Notably, GLM-130B's implementation is significantly faster than BLOOM-176B's Pytorch implementation, enhancing its usability in practical applications.

Activation Outlier Analysis

A unique aspect of GLM-130B is its handling of activation outliers in the quantization process. Unlike other GPT-based LLMs, GLM-130B presents a significant number of outlier dimensions (30%), posing challenges for standard quantization methods.

These outliers are crucial for the model's performance, containing potentially important language or world knowledge.

Weight Quantization Techniques

GLM-130B employs Absmax and Zeropoint Quantization methods for its linear layers, while maintaining the original precision for certain components.

The study explores the effects of these quantization methods at different scales, demonstrating GLM-130B's superior performance retention compared to BLOOM models, especially at INT4 precision.

Ablation Studies and Contribution Attribution

GLM-130B's architecture and training techniques undergo rigorous ablation studies to attribute their contributions to the model's overall performance.

The studies reveal that GLM's bidirectional attention and Multi-task Instruction Pre-training (MIP) are significant contributors, especially in tasks involving text similarity and coreference.

Key Lessons Learned

The GLM-130B project distills several key lessons, highlighting the importance of bidirectional architecture, platform-aware configuration, and improved post-normalization techniques.

It underscores the critical role of training stability, the scaling law of INT4 weight quantization unique to GLM, and future directions focusing on data quality, architectural improvements, and comprehensive training.

Advanced Technical Insights into GLM-130B's Model Training and Performance

Emergent Abilities in Large Language Models

GLM-130B demonstrates a fascinating phenomenon where certain task performances soar only after the model reaches a substantial size (e.g., 100B or 10B). This aligns with observations made in other LLMs like GPT-3, LaMDA, and PaLM. The BIG-bench benchmark is instrumental in collecting these emergent ability tasks. The intriguing scaling behaviors of these models open new avenues for research to understand the underlying mechanisms.

Comprehensive Training Configurations

Table 11 (In Paper) in the paper details the full spectrum of configurations used for training GLM-130B. This includes various parameters such as Adam optimization settings (beta values, epsilon), dropout rates, gradient clipping, and more. Notably, the model employs techniques like attention dropout, bias dropout fusion, and checkpoint activations. The configuration also specifies the use of FP16 precision, a learning rate schedule, and the GLU activation function, among others. These settings reflect the meticulous and fine-tuned approach to optimizing the model's training process.

Multi-task Instruction Pre-training (MIP) Datasets

Table 12 (In Paper) lists the 74 datasets involved in GLM-130B's Multi-task Instruction Pre-training. This extensive collection spans a wide range of tasks, from coreference resolution to sentiment analysis, extracting QA, and more. These datasets are instrumental in developing the model's diverse capabilities and ensuring its robustness across various language processing tasks.

Performance Comparison on BIG-bench-lite and MMLU

Table 14 (In Paper) presents a detailed comparison of GLM-130B with other prominent models like GPT-3 and PaLM on the BIG-bench-lite benchmark. This comparison, across different shots (0, 1, 3), showcases GLM-130B's capabilities in various tasks, highlighting its strengths and areas for improvement.

Table 15 (In paper) shows the performance of GLM-130B and BLOOM 176B on the MMLU benchmark. This comparison across various disciplines like STEM, Social Science, and Humanities provides a comprehensive view of the model's proficiency in different knowledge domains.

Technical Challenges and Project Management

The manuscript also reflects on the various technical and engineering challenges encountered during the project. It acknowledges the crucial role of student leaders, technical advisors, and the project leader in steering this complex and ambitious project to fruition

A Comprehensive Journey of GLM-130B: Challenges, Innovations, and Broader Impacts

The Genesis and Evolution of GLM-130B

The GLM-130B project began in December 2021 with the aim of creating a bilingual, highly accurate language model for both Chinese and English, addressing the limitations of GPT-3's availability and language support. The ambitious endeavor was not without its challenges, including computational resource constraints, the need for a robust pre-training algorithm for a bilingual model, and devising fast, low-resource inference solutions.

The choice of the GLM algorithm was pivotal due to its practical efficacy, and the decision to train a 130 billion parameter model was driven by the goal of enabling inference on a single A100 server.

Overcoming Technical Hurdles

The journey from inception to successful training was fraught with technical difficulties. Initial attempts in January 2022 quickly uncovered the complexities of training a model of this magnitude. Challenges ranged from hardware failures, gradient explosions, unexpected memory usage issues, to debugging pipeline parallelism in new frameworks. Collaborative efforts, particularly with the Tsinghua PACMAN team, were crucial in overcoming these obstacles.

By April 2022, further optimizations were made to adapt the model to various platforms, improving convergence and addressing new issues like large gradient norms and different layer normalization techniques. These adaptations eventually made the model runnable across multiple platforms.

Broader Impacts

AI Research: The release of GLM-130B democratizes access to large language models, enabling researchers to conduct in-depth studies, modify architectures, and test algorithms for improving LLMs.

Developers and Small Companies: GLM-130B offers a cost-effective solution for integrating LLMs into business applications, allowing for fine-tuning and distillation on their specific tasks.

Society: Addressing the potential misuse of LLMs, GLM-130B promotes inclusivity and transparency, enabling better defense against harm and deeper analysis of LLMs' flaws.

Environmental Considerations

A critical aspect of the GLM-130B project is its environmental impact. The training consumed 442.4 MWh of electricity, resulting in 257.01 metric tons of CO2 emissions, approximately half of what was estimated for GPT-3. This reduced footprint is attributed to efficient parallel strategies and hardware improvements. Releasing GLM-130B is seen as a step towards reducing further emissions associated with replicating similar-scale LLMs.