GPU Hardware and Software Ecosystem

This episode introduces the GPU hardware and software ecosystem, providing an overview of how modern GPUs are designed and how they are used for HPC and accelerated applications. It covers the key components of GPU hardware, including GPU architectures, processing units, memory hierarchy, and the differences between CPUs and GPUs.

This episode also discusses the software stack that enables GPU programming, including GPU platforms, libraries, and frameworks that allow applications to efficiently utilize GPU resources. Understanding the interaction between GPU hardware and software is essential for developing optimized GPU-accelerated applications.

Questions

• What are the main architectural differences between GPUs and CPUs, and why are GPUs suitable for parallel computing?

• What are the key components of GPU hardware, including processing units, memory hierarchy, and interconnects?

• What GPU software stacks are available, and what tools, libraries, and programming models do they provide?

• How do GPU hardware features and software tools work together to improve the performance of scientific and HPC applications?

Objectives

• Understand the fundamental differences between GPUs and CPUs and their roles in modern computing systems.

• Explore major GPU software ecosystems, such as CUDA, ROCm, and oneAPI, and gain a basic understanding of their components and capabilities.

• Learn about the key hardware components of GPUs, including processing units, memory hierarchy, and architecture design principles.

Instructor note

• 20 min teaching

• 10 min exercises/discussion

An Overview of GPU Hardware

A CPU is designed for general-purpose computing and typically consists of a small number of powerful, complex cores optimized for sequential execution, low-latency operations, and handling diverse workloads, as shown in the figure below. In contrast, a GPU contains thousands of smaller, simpler cores designed to execute many operations in parallel. GPU cores share control logic and efficiently process large amounts of data simultaneously, allowing many more cores to be integrated on a single chip and achieving much higher computational density.

While CPUs excel at tasks requiring complex decision-making and fast response times, GPUs are optimized for highly parallel workloads, such as scientific simulations, deep learning, and data-intensive applications.

Figure: A comparison of the CPU and GPU architecture.

GPUs, or general-purpose accelerators, are typically separate computing devices that contain their own processors, memory, power management components, and other supporting hardware. They are connected to the CPU and the main system motherboard through a high-speed interconnect, such as the PCIe bus. Traditionally, having dedicated GPU memory meant that data had to be explicitly transferred between the CPU and GPU before and after computation, although newer architectures increasingly support more unified and shared memory models.

The CPU acts as the host processor, managing the overall execution workflow. It prepares data, transfers it to the GPU when needed, launches GPU kernels, and retrieves the results after computation. In contrast, GPUs are designed to execute massive numbers of lightweight threads simultaneously across thousands of cores, while performing minimal control and data management. Because many threads may access memory concurrently and GPUs typically have limited cache compared to CPUs, inefficient memory usage can quickly become a performance bottleneck. Therefore, careful data organization and optimized memory access patterns are essential for achieving high GPU performance.

GPUs achieve high efficiency by keeping many threads active at the same time. They can rapidly switch between threads, allowing computation to continue while other threads wait for slower memory operations to complete. This ability to hide memory latency through massive parallelism is one of the key advantages of GPU architectures.

A key feature of GPU accelerators is their scalability. The computational cores in modern accelerators are typically organized into groups of multiprocessors, where processing elements share control logic, data paths, and other hardware resources. This design enables a very high density of compute units within a single GPU device. It also allows GPUs to scale efficiently: increasing the number of multiprocessors directly increases the available parallel computing resources and overall performance. As transistor density continues to improve, additional compute units can be integrated into GPU architectures, providing a straightforward path toward higher computational capability.

CPU	GPU
General-purpose processor designed for a wide range of workloads	Specialized processor optimized for massively parallel workloads
Optimized for serial processing and complex control operations	Optimized for parallel processing and large-scale data operations
Excels at task parallelism (running different tasks simultaneously)	Excels at data parallelism (performing the same operation on large datasets)
Designed for low latency and fast response per thread	Designed for high throughput and processing many threads simultaneously
Large portion of chip area dedicated to cache memory and control logic	Large number of compute units dedicated to arithmetic operations, especially floating-point calculations

In short

• GPU accelerators deliver high performance through their scalability and high density of compute elements.

• GPUs are typically separate circuit boards connected to CPUs via a PCIe bus and equipped with their own dedicated memory.

• CPU manages the execution workflow by transferring data from its memory to the GPU memory, launching computations, and copying results back.

• GPUs execute thousands of threads simultaneously and rapidly switch between them to hide memory latency and improve efficiency.

• Efficient data management and optimized memory access patterns are critical for achieving high GPU performance and avoiding memory bottlenecks.

GPU Programming Platforms and Software Stacks

GPUs are supported by software stacks and APIs that provide a standard interface between applications and GPU hardware. These tools allow developers to take advantage of GPU parallel processing capabilities by providing access to essential features such as memory management, CPU-GPU data transfers, and parallel task execution. They also include optimized libraries for common HPC workloads, such as linear algebra and fast Fourier transforms (FFT), as well as debugging and profiling tools to help developers optimize performance and ensure code correctness.

Major GPU vendors, including NVIDIA, AMD, and Intel, provide their own software ecosystems: CUDA, ROCm, and oneAPI, respectively. These platforms enable hardware-specific optimizations and provide specialized features for their architectures. In addition to vendor-specific solutions, cross-platform programming models such as OpenCL, SYCL, and DirectCompute provide more portable approaches for GPU programming across different hardware platforms.

CUDA

CUDA (Compute Unified Device Architecture) is NVIDIA’s parallel computing platform and programming model for GPU-accelerated applications. The CUDA API provides a comprehensive set of tools and interfaces that enable developers to create high-performance applications running on NVIDIA GPUs. It consists of two main components: the CUDA Toolkit and the CUDA Driver.

• The toolkit includes libraries, compilers, and development tools for programming and optimizing CUDA applications.

• The driver provides communication between the host CPU and the GPU device. CUDA supports widely used programming languages such as C/C++, and Fortran.

The CUDA ecosystem provides many highly optimized libraries for common HPC workloads, including cuBLAS for dense linear algebra operations (e.g., matrix multiplication), cuFFT for fast Fourier transforms, cuRAND for random number generation, and cuSPARSE for sparse matrix operations, and cuDNN for deep neural network primitives and accelerated deep learning operations. These libraries allow developers to accelerate complex computations without the need to implement low-level GPU kernels manually.

CUDA applications can be developed using several NVIDIA compilers. The primary compiler, nvcc, is based on the LLVM (low level virtual machine) compiler infrastructure and generates optimized code for NVIDIA GPUs while supporting host compilers for architectures such as AMD, Intel, OpenPOWER, and Arm CPUs. NVIDIA also provides additional compilers, including nvc (C11 compiler), nvc++ (C++17 compiler), and nvfortran (ISO Fortran 2003 compiler), which support GPU programming through parallel programming models such as OpenACC and OpenMP.

The CUDA Toolkit also includes debugging and correctness tools. cuda-gdb, an extension of the GNU Project Debugger (GDB), is a command line tool and enables simultaneous debugging of CPU and GPU code running on real hardware. This enables developers to debug applications directly on real GPU hardware, avoiding potential discrepancies introduced by simulation or emulation environments. In addition, the command line tool compute-sanitizer helps identify common GPU programming errors, including invalid memory accesses, buffer overflows, race conditions, and uninitialized variables.

To optimize application performance, NVIDIA provides profiling tools such as NVIDIA Nsight Systems and NVIDIA Nsight Compute.

• The Nsight Systems performs system-level analysis by collecting information about CPU and GPU utilization, memory transfers, and overall application behavior.

• The Nsight Compute provides detailed kernel-level analysis, including metrics related to execution efficiency, memory usage, instruction throughput, and occupancy.

Beyond these components, NVIDIA provides a broad range of additional libraries, tools, and frameworks, making CUDA one of the most mature and comprehensive GPU computing ecosystems available.

Figure: CUDA applications and libraries. Source: NVIDIA.

CUDA - In short

• CUDA: NVIDIA’s parallel computing platform

  • Components: CUDA Toolkit and CUDA Driver

  • Supports C/C++, and Fortran programming languages

• CUDA API Libraries: cuBLAS, cuFFT, cuRAND, cuSPARSE, cuDNN

  • Provide optimized routines for accelerating complex HPC and AI workloads on GPUs

• Compilers: nvc, nvcc, nvc++, nvfortran

  • Support GPU and multicore CPU programming

  • Compatible with parallel programming models such as OpenACC and OpenMP

• Debugging Tools: cuda-gdb, compute-sanitizer

  • Enable simultaneous debugging of GPU and CPU code

  • Detect memory access errors, race conditions, and other programming issues

• Performance Analysis Tools: NVIDIA Nsight Systems and NVIDIA Nsight Compute

  • Provide system-level and kernel-level performance analysis

  • Help optimize CPU/GPU utilization, memory bandwidth, and computational efficiency

• A mature and comprehensive CUDA ecosystem with extensive libraries, tools, and frameworks for GPU-accelerated computing.

ROCm

ROCm (Radeon Open Compute) is AMD’s open software platform for GPU-accelerated computing, enabling researchers and developers to utilize AMD accelerators. The ROCm platform is built on the foundation of open portability, supporting diverse accelerator architectures and environments. Similar to CUDA, ROCm provides a complete ecosystem of libraries, compilers, and development tools for programming and optimizing applications on AMD GPUs. It includes libraries, compilers, and development tools for programming and optimizing applications. For debugging, ROCm provides the command-line tool roc-gdb, while rocprof and roctracer enable performance analysis and profiling.

For GPU programming, ROCm uses the Heterogeneous-Computing Interface for Portability (HIP), a C++ runtime API and toolset that enables portable GPU applications across AMD and NVIDIA platforms. The hipcc compiler will call the appropriate toolchain depending on the desired platform.

• on AMD platforms, HIP provides a header and runtime library built on top of the HIP-Clang (ROCm compiler).

• On an NVIDIA platform, HIP provides a header file which translates from the HIP runtime APIs to CUDA runtime APIs. The header file contains mostly inlined functions and thus has very low overhead. The code is then compiled with nvcc, the standard C++ compiler provided with CUDA.

On AMD platforms, HIP libraries are typically prefixed with roc and can be called directly from HIP. For portable applications, HIP also provides hip-prefixed wrappers that map to the underlying vendor-specific implementations with virtually no performance overhead. These wrappers enable the same HIP code to run across different GPU platforms with minimal or no modifications. Most ROCm libraries have close counterparts in the CUDA ecosystem, providing nearly one-to-one functionality.

In addition, ROCm integrates with popular machine learning frameworks such as PyTorch and TensorFlow, providing optimized libraries and drivers to accelerate AI and machine learning workloads on AMD GPUs. This enables researchers to efficiently train and deploy machine learning models while taking full advantage of AMD accelerator hardware.

Figure: AMD ROCm ecosystem. Source: AMD.

ROCm - In short

• ROCm: Open software platform for AMD accelerators

  • Built around open portability across multiple vendors and architectures

  • Provides libraries, compilers, and development tools for AMD GPUs

  • Supports C/C++, and Fortran

  • Supports GPU and multicore CPU programming

• Debugging: roc-gdb

  • Command-line debugger for GPU applications

• Performance Analysis: rocprof and roctracer

  • Profile, analyze, and optimize application performance

• Programming Models

  • Supports HIP, OpenMP, and OpenCL

• Heterogeneous-Computing Interface for Portability (HIP)

  • Enables source portability across AMD and NVIDIA GPUs (Intel support is planned)

  • Provides hipcc compiler driver and runtime libraries

• GPU Libraries

  • AMD libraries are prefixed with roc and can be called directly from HIP

  • hip-prefixed wrappers provide portable interfaces with virtually no performance cost

oneAPI

Intel oneAPI is an open, unified software toolkit developed by Intel and it enables developers to build and optimize applications across a wide range of hardware, including CPUs, GPUs, and FPGAs. By providing a comprehensive set of tools, libraries, and frameworks based on open standards, oneAPI allows developers to write code once and deploy it on multiple architectures with minimal modifications, improving code portability, productivity, and performance.

The core of this ecosystem is the Intel oneAPI Base Toolkit, which provides compilers, optimized libraries, debugging tools, and performance analysis utilities for developing high-performance, data-centric applications across diverse architectures. It includes the DPC++/C++ Compiler (implementing SYCL), oneMKL (Math Kernel Library), oneDNN (Deep Neural Network Library), oneCCL (Collective Communications Library), oneTBB (Threading Building Blocks), Intel Distribution for GDB, Intel Advisor, Intel VTune Profiler, and several additional libraries and developer tools. For HPC applications, the Intel oneAPI HPC Toolkit extends the Base Toolkit with Intel MPI, Intel Fortran and C++ compilers, Cluster Checker, Inspector, and Intel Trace Analyzer and Collector.

oneAPI supports multiple programming models and languages for heterogeneous computing. It enables developers to write OpenMP applications for multicore CPUs and Intel GPUs using the Intel C++ and Fortran compilers, as well as SYCL applications for CPUs, GPUs, and FPGAs using the DPC++ compiler. While DPC++ originally targeted Intel GPUs through the oneAPI Level Zero backend, it now also supports NVIDIA GPUs (using CUDA) and AMD GPUs (using ROCm), enabling greater portability across hardware platforms.

Overall, Intel oneAPI provides a comprehensive and unified ecosystem for heterogeneous computing, enabling developers to optimize and deploy applications across diverse hardware architectures. By abstracting hardware-specific complexities and offering a consistent programming interface based on open standards, oneAPI promotes code portability, developer productivity, and performance across CPUs, GPUs, and other accelerators.

Figure: Intel’s oneAPI Base Toolkit. Source: Intel oneAPI.

oneAPI - In short

• Intel oneAPI: Unified software toolkit for developing and optimizing applications across heterogeneous architectures

  • Supports CPUs, GPUs, and FPGAs

  • Promotes code portability, reusability, and performance portability

• Intel oneAPI Base Toolkit: Core collection of compilers, libraries, and development tools

  • Includes the DPC++/C++ compiler with SYCL support

  • Provides libraries for communication, data analytics, deep learning, linear algebra, and more

• Intel oneAPI HPC Toolkit

  • Extends the Base Toolkit with Intel MPI, Intel C++ and Fortran compilers, debugging, and performance analysis tools

• Programming Models and Languages

  • Supports OpenMP, SYCL, C, C++, and Fortran

  • Applications written with open standards are portable across compatible implementations

• DPC++ Compiler

  • Supports Intel, NVIDIA, and AMD GPUs

  • Targets Intel GPUs through oneAPI Level Zero, NVIDIA GPUs through CUDA, and AMD GPUs through ROCm

• Debugging and Performance Analysis Tools

  • Intel Distribution for GDB, Intel Advisor, Intel VTune Profiler, Cluster Checker, Inspector, and Intel Trace Analyzer and Collector

• Open and Unified Ecosystem

  • Simplifies heterogeneous programming through a consistent programming interface

  • Improves developer productivity while enabling portable, high-performance applications across multiple hardware platforms

Differences and similarities

GPUs support different hardware features, even within the same vendor’s product lineup. In general, newer GPUs introduce additional capabilities, while some older features may be deprecated or removed. Consequently, it is important to compile binaries for the specific target architecture. A binary built for a newer GPU architecture will generally not run on older devices, whereas a binary built for an older architecture may execute correctly on newer GPUs but fail to take advantage of their full performance and capabilities.

In CUDA, the target compute capability is specified with the -arch=sm_XY compiler option, where X specifies the major architecture version (currently ranging from 1 to 9), and Y denotes the minor revision. When using HIP on NVIDIA platforms, the equivalent option is --gpu-architecture=sm_XY. On AMD platforms, the target architecture is specified as --offload-arch=gfxabc, where abc is the architecture identifier (for example, 90a for the MI200 series or 908 for the MI100 series).

For portable single-source applications, OpenMP target offloading allows the same source code to be compiled for both multicore CPUs and GPUs. In this case, the compilation specifies both the openmp target and the desired GPU architecture, enabling a single code base to execute efficiently across different hardware platforms.

Terminology

The terminology across NVIDIA, AMD, and Intel is not directly comparable because their GPU architectures differ. The following table provides an approximate mapping of commonly used terms and highlights the closest corresponding concepts across vendors.

GPU hardware terminology across vendors

NVIDIA	AMD	Intel
Thread	Work-item	Work-item / Thread
Warp	Wavefront	SIMD thread group (SIMD8/16/32)
Streaming processor (CUDA core)	SIMD lane (Stream processor)	Processing element (ALU)
SIMT unit	SIMD unit	Vector engine (XVE)
Streaming Multiprocessor (SM)	Compute Unit (CU)	Xe-core / Execution unit (EU)
Graphics Processing Cluster (GPC)	Shader Engine / Compute Engine	Xe Slice

Warning

The terminology in this table is only an approximate correspondence. GPU architectures differ significantly between vendors, and there is generally no one-to-one mapping between the terms used by NVIDIA, AMD, and Intel. Similar names often refer to components with different designs, capabilities, or execution models.

Summary

• GPUs are designed to execute thousands of threads concurrently, making them highly parallel processors. In contrast, CPUs are optimized for executing a smaller number of threads with greater emphasis on low-latency and complex control operations.

• GPUs dedicate a larger fraction of their transistors to arithmetic and data processing units rather than to data caching and flow control. This design prioritizes throughput-oriented computing, enabling GPUs to efficiently perform massively parallel workloads and hide memory access latency through concurrent execution.

• GPU vendors provide comprehensive software ecosystems, including toolkits, libraries, and compilers, to enable the development of high-performance applications that exploit GPU parallelism. Examples include CUDA (NVIDIA), ROCm (AMD), and oneAPI (Intel).

• These ecosystems also provide debugging and profiling tools, such as cuda-gdb and roc-gdb for debugging, and NVIDIA Nsight Systems, NVIDIA Nsight Compute, rocprof, and roctracer for performance analysis. These tools help developers optimize applications and maximize GPU resource utilization.

Exercises

Discussion about GPU usage and software ecosystem experience

1. Do you currently have access to GPUs for your research, studies, or projects?

• If yes, which GPU vendor do you primarily use?

  • NVIDIA GPU? AMD GPU? Intel GPU? Other?

• What GPU model or architecture do you use (if known)?

2. What GPU programming frameworks or software environments have you used?

• CUDA? ROCm/HIP? Intel oneAPI/SYCL?

• OpenMP target offloading?

• Other frameworks?

3. What programming languages or models do you use for GPU acceleration (e.g., C/C++, Fortran, Python, OpenMP, MPI, SYCL)?

4. Which GPU libraries or frameworks have you used (e.g., BLAS, FFT, machine learning frameworks, simulation libraries)?

GPU programming ecosystems

Which statement about GPU programming frameworks is true?

• A. All GPU vendors use the same programming framework and compiler tools.

• B. Applications written for one GPU vendor will always run efficiently on GPUs from other vendors without modification.

• C. NVIDIA, AMD, and Intel provide different software ecosystems (such as CUDA, ROCm, and oneAPI) to support their GPU architectures.

• D. GPU programming does not require specialized libraries, compilers, or performance analysis tools.

Solution

The correct answer is C). Different GPU vendors design their hardware differently, so they provide their own software tools, compilers, and libraries to allow developers to use the capabilities of their GPUs effectively.

GPU architecture and performance

Which statement about GPU architecture and performance is true?

• A. GPUs achieve high performance mainly by increasing the clock frequency of a small number of powerful cores, similar to CPUs.

• B. GPUs are optimized for sequential tasks with complex branching and low parallelism.

• C. A GPU application automatically achieves maximum performance without considering the target GPU architecture.

• D. GPUs are designed to execute many threads concurrently, allowing them to efficiently handle highly parallel workloads.

Solution

The correct answer is D). GPUs contain thousands of smaller processing units that can execute many threads at the same time. This massive parallelism makes GPUs well suited for tasks where the same operation is applied to large amounts of data, such as scientific simulations, machine learning, and data processing. Unlike CPUs, GPUs focus on high throughput rather than executing a small number of complex tasks quickly.

GPUs and memory

Which statement about the relationship between GPUs and memory is true?

• A. GPUs are not affected by memory access latencies.

• B. GPUs can run out of memory quickly with many cores trying to access the memory simultaneously.

• C. GPUs have an unlimited cache size.

• D. GPUs prefer to run with a minimal number of threads to manage memory effectively.

Solution

The correct answer is B). This is true because GPUs run many threads simultaneously on thousands of cores, and with limited cache available, this can lead to the GPU running out of memory quickly if many cores are trying to access the memory simultaneously. This is why data management and access patterns are essential in GPU computing.

Keypoints

• GPUs vs. CPUs: Key architectural and performance differences, including parallelism, execution models, and suitable workloads.

• GPU software ecosystems: Overview of vendor-specific software suites, their support for GPU features, programming models, and hardware compatibility.

• GPU applications: Common use cases of GPUs in scientific computing, artificial intelligence, data processing, and other high-performance computing workloads.