What are the most recent developments in parallel programming for GPUs?

1 CUDA 11.4

CUDA 11.4, released in June 2021, is the latest version of the popular platform and programming model for GPU-accelerated applications. This version introduces several new features and improvements, such as support for NVIDIA Ampere architecture GPUs, which offer higher performance and lower power consumption. Additionally, CUDA 11.4 enhances interoperability with other libraries and frameworks, such as cuDNN, cuBLAS, cuSPARSE, cuFFT, and TensorFlow. It also includes improved debugging and profiling tools like Nsight Compute, Nsight Systems, Nsight Graphics, and Nsight Visual Studio Edition. Finally, new extensions and APIs for managing memory, streams, events, graphs, and modules are available.

2 SYCL 2020

SYCL is a cross-platform abstraction layer that enables parallel programming for heterogeneous devices, such as CPUs, GPUs, FPGAs, and DSPs. Built on the C++ standard and supporting modern C++ features like lambdas, templates, and exceptions, SYCL 2020 was released in November 2020 with several new features and enhancements. These include unified shared memory (USM) for simplified memory management and data movement, reductions for efficient parallel operations, sub-groups for finer-grained control over work-items within a work-group, atomic operations for concurrent access of shared data without race conditions, and interoperability with other APIs like OpenCL, Vulkan, and CUDA.

3 HPX 1.7

HPX is a general-purpose C++ library that provides a unified programming model for parallel and distributed computing, based on the concept of futures and continuations. The latest version, HPX 1.7 released in June 2021, introduces several new features and improvements. These include support for NVIDIA GPUs, integration with Kokkos, improved performance and scalability for large-scale applications and systems, as well as new modules and algorithms such as dataflow, resiliency, sort, and scan.

4 Kokkos 3.4

Kokkos 3.4, released in May 2021, is the latest version of the C++ library that enables performance-portable parallel programming for various architectures, such as CPUs, GPUs, and FPGAs. This version introduces several new features and improvements, such as support for AMD GPUs which expands the range of devices that Kokkos can target, improved interoperability with other libraries and frameworks like SYCL, MPI, and OpenMP, enhanced debugging and testing tools like Kokkos Tools, Kokkos Profiling, and Kokkos Unit Test, as well as new features and optimizations like hierarchical parallelism, team policies, scatter views, and atomic views.

5 Julia 1.6

Julia is a high-level, dynamic, and expressive language that combines the ease of use of scripting languages with the performance of compiled languages. It is designed for scientific and numerical computing, and supports multiple dispatch, metaprogramming, and parallelism. The latest version of Julia (1.6, released in March 2021) introduces several new features and improvements, such as support for NVIDIA CUDA GPUs which enables Julia to run GPU kernels and access GPU libraries, improved multithreading which allows Julia to leverage multiple CPU cores and threads for parallel computation, faster compilation and loading which reduces the latency and overhead of starting and running Julia programs, as well as new libraries and packages like Flux, DifferentialEquations, Plots, and DataFrames.

6 Rust-GPU 0.1

Rust is a low-level, safe, and concurrent language that aims to provide memory safety, zero-cost abstractions, and fearless concurrency. It is widely used for system programming, web development, and embedded applications. Rust-GPU is a project that enables Rust to compile and run on GPUs, using SPIR-V as the intermediate representation. The first alpha release of Rust-GPU 0.1 was released in October 2020 and introduced several new features and improvements. These include support for basic GPU features such as attributes, uniforms, textures, samplers, and buffers; integration with other tools and frameworks such as wgpu, naga, and shaderc; improved documentation and examples like Mandelbrot, Conway's Game of Life, and Ray Tracing; and new macros and attributes like #[spirv(entry_point)], #[spirv(storage_class)], and #[spirv(glsl)].

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?