Accelerating Generative AI: NVIDIA's CUDA Reinvents HPC

By Bamiyan Gobets

Go catch a 'Cuda... if you can?

英伟达 's CUDA, originally associated with the Plymouth Barracuda (‘CUDA), embodies the spirit of raw power, speed, elegance, and unrestrained innovation. ?

CUDA (Compute Unified Device Architecture) has enabled and redefined traditional HPC (high performance computing) capabilities and infrastructure, enabling both traditional and modern applications to utilize GPU computing infrastructures with extreme efficiency and power compared to traditional CPU-only based infrastructures of the past. ?

The origin of CUDA

In 2003, a team of researchers led by Ian Buck unveiled Brook, the first widely adopted programming model to extend C with data-parallel constructs. Buck later joined NVIDIA and led the launch of CUDA in 2006, the first commercial solution for general purpose computing on GPUs.

10,000 Feet

At the highest level, NVIDIA's CUDA is a parallel computing (enabling) platform and an API (application programming interface). ?

CUDA allows software developers to use one or more (parallelized) NVIDIA GPUs (graphics processing units) for highly optimized, general-purpose processing. ?

Essentially, CUDA grants direct access to the virtual instruction set and computational elements of the GPU, thereby enabling dramatic increases in computing performance by harnessing the power of the graphics processing unit to accelerate computational tasks traditionally handled by the central processing unit (CPU). ?

CUDA has become a key tool in the domain of modern HPC (high-performance computing), machine learning, deep learning, scientific simulations, and graphical rendering, providing a bedrock solution for parallelizing almost any traditional CPU based, or modern GPU designed workloads. ?

CUDA provides an entire ecosystem of support with toolkits, libraries, and support for various OSs to ensure its wide adoption and use in the industry. ?

Quick Refresher - CPUs VS GPUs?

HPC has evolved to be strongly associated with executing parallel computing across GPUs, which are designed to enhance computing parallelism, efficiency, and performance far beyond what is feasible with traditional CPUs.?

1. While a CPU can handle multiple operations (somewhat) concurrently when utilizing multiple cores, it is fundamentally optimized for sequential tasks and scalar operations.?
2. In contrast, a GPU is built specifically to execute operations in parallel, making it particularly suited for vector and matrix multiplications, which are foundational operations in graphics rendering and deep learning algorithms.?

CUDA has catalyzed the next wave of innovation in parallel computing, fostering a platform that enables applications to fully harness the immense power of computational parallelism across numerous GPUs.?

CUDA Enables HPC for (Generative) AI and Data Science Workloads

Today we associate ‘AI workloads’ primarily with Generative AI (and Data Science). Generative AI refers to?models or algorithms that create brand-new output, such as text, images, videos, code, data, or 3D renderings, from the vast amounts of data they are trained on.??

Some popular real-world examples of Generative AI are ChatGPT, Dall-E and Midjourney. These interfaces use LLMs (Large Language Models) to create new output based on the data they’ve been trained on. These applications are just examples which help us understand the potential of such technology within the enterprise.

Imagine, someday most corporate data will be leveraged to generate AI output in all forms allowing users to communicate with data in (Human) Natural Language. It’s a huge leap from where we are today. ?

Simple Generative AI Pipeline and Workload Example – LLM

While GPUs can be used in the data collection stage (for example to accelerate Spark with CUDA and RAPIDS by 5-10x). The most significant bottlenecks in the Model Training and Inference stages where periodic and ongoing computational power defines the performance and user experience. Especially in the training phase, leveraging CUDA and GPU distribution can reduce training time from (for example) 1 year to several days or weeks. ?

As Inference (asking questions of your data and receiving output) will need to be constantly updated and for users to receive ‘fast’ responses, ongoing GPUs will be utilized. Inference models are getting bigger as more data can be processed, so computational challenges will persist as Inference runtime matures and new boundaries are explored. ?

What Does Cuda Actually Do?

The CUDA programming model allows scaling software transparently with an increasing number of processor cores in GPUs. You can program applications using CUDA language abstractions. Any problem or application can be divided into small independent problems and solved independently among these CUDA blocks. Each CUDA block offers to solve a sub-problem into finer pieces with parallel threads executing and cooperating with each other. ?

The CUDA runtime decides to schedule these CUDA blocks on multiprocessors in a GPU in any order. This allows the CUDA program to scale and run on any number of multiprocessors.??

Figure (ABOVE) shows this massive multiprocessor scaling architecture. The compiled CUDA program has eight CUDA blocks. The CUDA runtime can choose how to allocate these blocks to multiprocessors as shown with streaming multiprocessors (SMs).?For a smaller GPU with four SMs, each SM gets two CUDA blocks. For a larger GPU with eight SMs, each SM gets one CUDA block. This enables performance scalability for applications with more powerful GPUs without any code changes.?

2023 - NVIDIA Taking the Generative AI Market by Storm

CUDA is the core software enabler of NVIDIA's hardware strategy.

In recent times, CUDA has firmly established itself as a pivotal catalyst, propelling both classical and contemporary applications to harness the boundless parallel computing capabilities, a testament to NVIDIA’s groundbreaking GPU innovations and its stronghold on the market.?

Standing at the nexus of a technological revolution, CUDA is facilitating the exponential growth and potential of Generative AI, a force that is fundamentally reshaping the contours of our global landscape with advancements that have become intrinsic to our daily experiences. When it comes to revenue generated from Data Center operations in the realm of artificial intelligence, NVIDIA's trajectory is on a swift ascent, significantly outpacing rivals like AMD and Intel.?

NVIDIA is leagues ahead of its competition. With a commanding presence in the market, NVIDIA has garnered an impressive 70% market share in the Generative AI GPU sector, a margin that continues to widen.?

Representing a stronghold in enterprise operations, data center revenue is a testament to a robust foundation in machine learning and Generative AI, technologies that are rapidly becoming ubiquitous across a multitude of industries.??

The CUDA (Support) Ecosystem ?

CUDA’s essential objective as a software platform is to accelerate time-to-solution by dividing work across parallel Tensor Cores on a GPU and then (when needed) across multiple GPUs. ?

CUDA enables massively parallel computing operations vs traditional (multi-core) CPU enabling NVIDIA’s customers to move faster than they ever could before. ?

Let’s explore the key CUDA Ecosystem components here with examples ?

(1.) CUDA OS Support?

CUDA supports popular Linux and Windows OSs for developer access. ?

(2.) CUDA Toolkits and Drivers

A Few Examples: ?

• CUDA Compiler: transformation of CUDA code written in programming languages like C, C++, and Fortran into a format that can be understood by the GPU.?
• Developer Tools: profiling, debugging, and optimizing the performance of CUDA applications. These include the NVIDIA Nsight suite and the Visual Profiler?
• C++ Core: CUDA extends the C++ programming language, allowing for the straightforward integration of parallel computing concepts within a familiar programming context. ?
• Memory Management: CUDA offers fine-grained control over memory management, empowering developers to optimize data transfers and allocation. Includes the management of shared memory, constant memory, and registers.?
• Windows & Graphics Integration: CUDA seamlessly integrates with Windows operating systems. ?
• Comms Libraries: These are a collection of highly optimized communication libraries that facilitate efficient data exchanges in multi-GPU and distributed computing environments. These libraries are critical for building scalable parallel applications that require high-speed communication between different computing nodes.?

(3.) CUDA-X Libraries

CUDA-X, built on top of NVIDIA CUDA, is a collection of libraries that deliver more specific support and higher performance for specific domains and workloads like artificial intelligence (AI) and HPC:?

A Few Examples:?

Math?

• cuBLAS: GPU-accelerated basic linear algebra (BLAS) library?
• cuFFT: GPU-accelerated library for Fast Fourier Transforms?
• CUDA Math Library: GPU-accelerated standard mathematical function library?
• cuRAND: GPU-accelerated random number generation (RNG)?

Parallel Algorithms?

• THRUST: GPU-accelerated library of C++ parallel algorithms and data structures?
• cuLITHO: Library with optimized tools and algorithms to GPU-accelerate computational lithography and the manufacturing of semiconductors?

Image and Video ?

• NvJPEG: High performance GPU-accelerated library for JPEG decoding?
• Performance Primitives: Provides GPU-accelerated image, video, and signal processing functions?
• Video Codec SDK: A complete set of APIs, samples, and documentation for hardware-accelerated video encode and decode on Windows and Linux?

Deep Learning?

• cuDNN: GPU-accelerated library of primitives for deep neural networks?
• TensorRT: High-performance deep learning inference optimizer and runtime for production deployment?
• RIVA: Platform for developing engaging and contextual AI-powered conversation app?
• DeepStream SDK: Real-time streaming analytics toolkit for AI-based video understanding and multi-sensor processing?
• DALI: Portable, open-source library for decoding and augmenting images and videos to accelerate deep learning applications?

Communication ?

• NVSHMEM: OpenSHMEM standard for GPU memory, with extensions for improved performance on GPUs.?
• NCCL: Open-source library for fast multi-GPU, multi-node communications that maximizes bandwidth while maintaining low latency.?

Partner Libraries?

• OpenCV: GPU-accelerated open-source library for computer vision, image processing, and machine learning, now supporting real-time operation?
• FFMPEG: Open-source multimedia framework with a library of plugins for audio and video processing?
• ArrayFire: GPU-accelerated open-source library for matrix, signal, and image processing?
• MAGMA: GPU-accelerated linear algebra routines for heterogeneous architectures?
• GUNROCK: Library for graph-processing designed specifically for the GPU?

(4.) CUDA Applications and Frameworks?

A Few Examples:

• TensorFlow: A popular open-source machine learning framework that can leverage CUDA for accelerated computation, particularly beneficial in deep learning tasks.?
• RAPIDS: A suite of software libraries and tools for executing end-to-end data science and analytics pipelines entirely on GPUs, typically using Python. Rapids utilizes CUDA for high-speed computations.?
• PyTorch: Another widely used open-source machine learning framework that integrates with CUDA to offer accelerated matrix operations and other computations, facilitating faster model training and inference.?
• Mxnet: A flexible and efficient library for deep learning that supports an array of languages and can utilize CUDA to enhance the speed of its operations, particularly in training deep neural networks.?
• Chainer: A Python-based deep learning framework that can employ CUDA to speed up its computations, offering a flexible and intuitive platform for neural networks and other machine learning tasks.?

Real-World Example of CUDA’s Benefits?

NVIDIA RAPIDS?- Underpinned by CUDA

RAPIDS, leveraging CUDA, is an open-source suite of GPU-accelerated Python libraries designed to improve data science and analytics pipelines. With APIs similar to popular open-source data science tools, RAPIDS uses NVIDIA CUDA primitives for low-level compute optimization. This provides access to GPU parallelism and high-bandwidth memory speed through Python interfaces, leading to a faster performance at scale across data pipelines.?

RAPIDS Suite:

• cuDF: cuDF is a GPU DataFrame library that provides a pandas-like API for loading, filtering, and manipulating data.?

• Spark-RAPIDS: The RAPIDS Accelerator for Apache Spark provides a set of plug-ins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries.?
• cuGRAPH: cuGraph is a GPU-accelerated graph analytics library that includes support for property graphs, remote (graph as a service) operations, and graph neural networks.?
• cuML: cuML is a suite of libraries that implements machine learning algorithms and mathematical primitives functions that share compatible APIs with other RAPIDS projects and matches APIs from scikit-learn in most cases.?

RAPIDS Accelerator for Spark 3?

NVIDIA has created a RAPIDS Accelerator for Apache Spark 3 that intercepts and accelerates extract, transform and load pipelines by dramatically improving the performance of Spark SQL and DataFrame operations.?

For reference, this can also be leveraged, for example by Databricks if deploying on NVIDIA GPU infrastructure. Seems like a good idea! It's well known that the original founders of Spark founded Databricks, and Databricks relies heavily on Spark!

Modifications to Spark Components?

Spark 3 provides columnar processing support in the Catalyst query optimizer, which is what the RAPIDS Accelerator plugs into to accelerate SQL and DataFrame operators. When the query plan is executed, those operators can then be run on GPUs within the Spark cluster.?

NVIDIA has also created a new Spark shuffle implementation that optimizes the data transfer between Spark processes. This shuffle implementation is built on GPU-accelerated communication libraries, including UCX, RDMA, and NCCL.?

GPU-Aware Scheduling in Spark?

Spark 3 recognizes GPUs as a first-class resource along with CPU and system memory.

This allows Spark 3 to place GPU-accelerated workloads directly onto servers containing the necessary GPU resources as they are needed to accelerate and complete a job.?

NVIDIA engineers have contributed to this major Spark enhancement, enabling the launch of Spark applications on GPU resources in Spark standalone, YARN, and Kubernetes clusters.?

A Real-World Use-Case with RAPIDS leveraging CUDA and SPARK 3

AT&T

AT&T applied the NVIDIA RAPIDS Accelerator for Apache Spark on GPU clusters for extract, transform, and load (ETL) and feature engineering stages in their data-to-AI pipeline, improving performance, reducing costs, and increasing simplicity compared to CPU-based Spark clusters and Databricks' Photon engine.

Wrapping-Up

Traditional HPC is Losing Steam?

Parallel computing has historically been the backbone of high-performance computing (HPC), typically operating on networks of single-threaded CPUs spanning hundreds or even thousands of nodes. However, conventional multi-core CPUs fall short in fueling the performance efficiency demanded by contemporary data processing models, which now handle data sets characterized by billions or even trillions of parameters.?

Even if you were happy with the performance CPUs could deliver, the economics at scale fail to make sense. Both in cost, energy consumption and effort to build (more) nodes to achieve the same result.

Modern HPC Reinvented by NVIDIA and Underpinned by CUDA. ?

CUDA, enabling NVIDIA’s simultaneous access to its own parallelized Tensor Cores, has proven pivotal in navigating this computational leap. Tensor Cores excel at orchestrating parallelized mixed-precision matrix-multiplication-and-accumulate operations, a cornerstone procedure in deep learning algorithms, significantly curtailing the time required to execute large-scale calculations.??

Final Words..

Some Enterprise Challenges NOT addressed by our Hero CUDA.?

Corporate Enterprises stand on the cusp of a groundbreaking shift. Recognizing the vast and untapped potential embedded in their data, organizations are leveraging new tools and approaches, poised to deliver sophisticated interfaces and enriched user experiences in the era of Large Language Models (LLMs), Generative AI, Data Science and Machine Learning.

Those who venture boldly into this new frontier of innovation are not only reaping rewards but also paving the way for a novel paradigm of business intelligence and operational excellence. Indeed, the signs of burgeoning success are already perceptible in the market for those who dare.

Yet, the trajectory to full transformation is characterized by a measured pace due to lack of experience. The majority of enterprise organizations still opt for a careful linear progression, keen on formulating strategies that proficiently address the multifaceted risks associated with such substantial change.??

Web: nebul.com

Contact: [email protected]

?