Why GPUs for High-Performance Computing?¶
This episode introduces the evolution of GPU computing, starting from the Moore’s law and the challenges of increasing computational demands. It explores the concept of parallel computing, explains the differences between CPUs and GPUs, and examines why GPUs have become essential for applications such as artificial intelligence, scientific computing, and high-performance computing.
The episode also looks at the growing role of GPUs in modern supercomputers through the TOP500 list, while discussing their limitations and trade-offs, including programming complexity, memory constraints, energy consumption, and cost. Learners will gain a balanced understanding of both the strengths and challenges of GPU-based computing.
Questions
What is Moore’s Law, and how has it influenced the evolution of modern computing?
What is parallel computing, and why is it important for solving increasingly complex computational problems?
What are the key differences between CPUs and GPUs, and why are GPUs widely used for HPC and AI workloads?
What are the advantages and limitations of GPU computing, and when might a CPU-based approach still be preferable?
Objectives
Understand the evolution of computing, including Moore’s Law and the shift toward parallel computing.
Explain the principles of parallel computing and workload distribution.
Compare CPUs and GPUs and understand their applications in modern computing.
Evaluate the benefits, limitations, and challenges of GPU computing.
Instructor note
15 min teaching
5 min exercises/discussion
Moore’s Law¶
The development of modern computers is intrinsically related to the number of transistors that can be integrated into semiconductor chips, particularly within central processing units (CPUs). As transistor technology has advanced, engineers have been able to build smaller, faster, and more energy-efficient processors, enabling significant improvements in computing performance. The continuous increase in transistor density has driven major breakthroughs in computer systems, from early personal computers to today’s powerful processors used in artificial intelligence, scientific computing, and large-scale data processing.
This trend is commonly described by Moore’s law, which states that the number of transistors on an integrated circuit roughly doubles every two years, leading to a significant increase in computing capability while reducing the cost per transistor. First observed by Gordon Moore in 1965, this prediction successfully described the rapid growth of semiconductor technology for several decades, particularly from the 1970s to the 2000s, as shown in the figure below. The continuous scaling of transistors allowed computers to become smaller, faster, and more affordable, shaping the evolution of modern digital technologies.
Note
Moore’s Law is not a scientific law based on a fundamental physical principle. It is an empirical observation describing the historical trend of increasing transistor density in integrated circuits. Its continuation has depended on continuous advances in semiconductor manufacturing, materials, and processor design.
Figure: The evolution of microprocessors. The number of transistors per chip has approximately doubled every two years. Before 2000s, the growth in single-core clock frequency was the primary driver of processor performance gains. However, further performance improvements can no longer be achieved solely by increasing CPU clock frequency due to power consumption and thermal limitations. Therefore around the mid-2000s, the industry shifted toward multi-core processor architectures to improve performance through parallel execution.¶
As more transistors are embedded into an integrated circuit, each individual transistor must become smaller, enabling shorter switching distances and faster signal transmission. This reduction in transistor size allows processors to operate at higher clock frequencies and improves CPU performance by increasing computational speed and efficiency. However, as transistor density continues to increase within a limited area, challenges related to power consumption, heat generation, and energy efficiency become increasingly significant.
The growing difficulty of managing thermal and power constraints has created a major barrier to improving CPU performance through traditional transistor scaling alone. This limitation is clearly illustrated in the figure below, where the growth of CPU clock frequency began to slow around the 2010s, marking a shift away from simply increasing frequency toward alternative approaches.
Therefore, improvements in CPU performance have historically been achieved through two main strategies over the years:
Increase the performance of a single processor core. This approach is primarily achieved by increasing clock frequency, enhancing microarchitecture, and optimizing instruction execution to improve the efficiency and speed of individual cores.
Increase the number of physical cores. This approach enables multiple tasks or computational operations to be executed simultaneously through parallel processing, improving overall performance for workloads that can be divided and processed concurrently.
Note
While single-core performance improvements dominated earlier generations of processors, limitations in power consumption and heat dissipation have shifted the focus toward multi-core architectures as a key pathway for continued performance growth.
Parallel Computing¶
Parallel computing is an approach to computation where a large and complex problem is divided into smaller, independent subtasks that can be processed concurrently, as shown in the figure below. Instead of executing instructions sequentially on a single processing unit, parallel computing distributes the workload across multiple processing units, allowing different parts of the problem to be solved simultaneously. This can significantly reduce execution time and improve computational efficiency, especially for large-scale and data-intensive applications.
Figure: Computing in parallel.¶
The fundamental idea behind parallel computing is to identify opportunities for dividing a problem into smaller subtasks that can be executed at the same time. For example, a large dataset can be separated into multiple smaller datasets, with each processing unit performing calculations on a different portion of the data. Once the individual tasks are completed, their results can be combined to produce the final solution.
Warning
Splitting a problem into multiple subtasks introduces overhead, including task coordination, communication between processing units, and synchronization. If the communication cost is too high, the performance improvement from parallel execution may be reduced or even eliminated.
However, the way a computational problem is divided into subtasks depends strongly on the characteristics of the problem itself. Some problems can be easily decomposed into independent tasks, while others require frequent communication and synchronization between processing units. Efficient parallelization therefore requires careful consideration of data dependencies, workload balance, and communication overhead.
To take advantage of parallel computing, various programming models and parallelization strategies have been developed. These include approaches such as task parallelism, where different tasks are executed simultaneously, and data parallelism, where the same operation is applied to multiple data elements at once. The choice of programming approach depends on the hardware architecture and the nature of the application, which plays a critical role in achieving high performance on modern computing systems.
Tip
Efficient parallel computing requires selecting an appropriate parallelization strategy.
Data parallelism is effective when same operation needs to be applied to large amounts of independent data.
Task parallelism is useful when different tasks can be executed concurrently.
Graphics Processing Units¶
Graphics Processing Units (GPUs) have become one of the most widely used hardware accelerators in recent years, and the term GPU is sometimes used interchangeably with accelerator in the context of high-performance computing. GPUs were originally designed to efficiently handle highly parallel workloads in computer graphics, such as image rendering and visual effects. However, due to their ability to perform a large number of simple operations simultaneously, GPUs have increasingly been adopted for scientific computing and HPC applications.
GPUs are specialized parallel processors designed to perform large numbers of floating-point operations efficiently. They typically act as co-processors that work alongside traditional CPUs: the CPU remains responsible for controlling the overall program workflow, managing system operations, and assigning tasks, while the GPU accelerates computationally intensive and highly parallel portions of the workload. By using many smaller processing units organized in a parallel architecture, GPUs can effectively utilize the increasing transistor density enabled by modern semiconductor technologies and deliver significantly higher computational throughput compared with conventional CPUs for suitable applications.
The use of GPUs can provide extremely high performance per computing node, allowing a single GPU-equipped workstation or server to outperform much larger CPU-only systems for certain types of workloads. This advantage has made GPUs a key technology in modern supercomputing systems and platforms. However, achieving these performance gains requires software designed to exploit GPU parallelism. Existing CPU-based programs often need significant modifications or complete redesigns to adopt GPU programming models, introducing additional challenges in terms of programming complexity, memory management, and optimization.
Host vs. device
GPU-enabled systems require a heterogeneous programming model that involves both the CPU and GPU working together. In this model, the CPU and its associated memory are typically referred to as the host, while the GPU and its dedicated memory are referred to as the device.
Below is a representative system architecture showing the connection between the CPU (host) and the GPU (device) through a PCI Express (PCIe) link. In a GPU-enabled system, the CPU and GPU have their own dedicated memory spaces and communicate by transferring data through the interconnect. The PCIe link provides the communication pathway between the host and device; however, its bandwidth is typically lower than the internal memory bandwidth of the GPU. For example, modern CPU memory bandwidth is commonly in the range of 60-100 GB/s, while GPU memory bandwidth can reach approximately 150-300 GB/s or higher, depending on the GPU architecture. However, the data transfer bandwidth between the CPU and GPU through the PCIe link is significantly lower, typically around 6–12 GB/s depending on the PCIe generation and configuration. Such a substantial difference highlights the importance of minimizing data movement between the CPU and GPU and keeping data on the GPU whenever possible to achieve maximum performance. Efficient GPU programming therefore focuses not only on maximizing computation on the device but also on reducing communication overhead between the host and device.
Figure: The connection between CPU and GPU via PCIe link. Figure is adapted from the Carpentry lesson GPU Programming.¶
Discussion about your experience with HPC resources
Do you use HPC resources for your research or projects?
If yes, what type of HPC system do you use?
Is it a GPU-accelerated HPC system?
Is it a CPU-only HPC system? How many CPU cores do you usually request or use?
What types of workloads or applications do you run on the system? For example, molecular simulations, computational fluid dynamics modeling, machine learning, or other scientific and engineering applications?
If no, what computing resources do you typically use for your regular computations?
Why Use GPUs for Computing?¶
GPUs have become an essential component of modern HPC systems because they offer significantly higher computational throughput than traditional CPUs for many data-parallel applications. By integrating thousands of lightweight processing cores on a single chip, GPUs can execute a large number of operations simultaneously, making them particularly well suited for scientific computing, image processing, and large-scale numerical simulations. As a result, GPU acceleration has become a key technology in both research laboratories and industrial computing environments.
High performance: GPUs can dramatically accelerate computational workloads that exhibit a high degree of parallelism. Applications such as matrix operations, molecular dynamics, computational fluid dynamics, and weather forecasting can often achieve substantial speedups compared with CPU-only implementations. For these workloads, a single GPU may deliver performance comparable to dozens or even hundreds of CPU cores.
Improved energy efficiency: GPUs are designed to perform many floating-point operations concurrently, allowing them to deliver significantly more computations per watt of power consumed than traditional CPUs. This improved energy efficiency is particularly important for modern supercomputers, where power consumption is a major operational cost. The increasing adoption of GPU-accelerated systems is reflected in the GREEN500 ranking, which evaluates the world’s most energy-efficient supercomputers.
Cost-effectiveness: Because of their high computational density, GPUs can provide excellent performance at a relatively low hardware cost for suitable applications. A single GPU-equipped workstation or server can often replace multiple CPU-only servers for parallel workloads, reducing hardware, maintenance, and energy costs. This makes GPU computing an attractive solution for both research institutions and commercial organizations.
The TOP500 List¶
The TOP500 project ranks and tracks the world’s 500 most powerful supercomputers based on their performance on the LINPACK benchmark. Established in 1993, the project publishes updated rankings twice each year and serves as one of the most widely recognized indicators of advances in HPC domains.
The snapshot below shows the top five supercomputers from the June 2026 TOP500 list. The table includes the following key performance metrics:
Cores – Total number of processor cores available in the system.
Rmax (PFlop/s) – The maximum performance achieved on the LINPACK benchmark.
Rpeak (PFlop/s) – The theoretical peak computational performance of the system.
Power (kW) – The system’s power consumption during benchmark execution.
Figure: Snapshot of the top five supercomputers from the TOP500 list (June 2026).¶
The current No. 1 system on the TOP500 list is LineShine, located at the National Supercomputing Center in Shenzhen, China. Unlike most of today’s leading supercomputers, LineShine achieves its performance using a CPU-only architecture without GPU accelerators. It is the first supercomputer to sustain more than 2 exaflops on the High Performance LINPACK benchmark.
A notable trend is that four of the five supercomputers are GPU-accelerated, including El Capitan (hosted by Lawrence Livermore National Laboratory (LLNL) in the United States), Frontier (hosted by Oak Ridge National Laboratory (ORNL) in the United States), Aurora (hosted by Argonne National Laboratory (ANL) in the United States), and JUPITER Booster (hosted by the Jülich Supercomputing Centre (JSC) at Forschungszentrum Jülich in Germany), using accelerators from AMD, Intel, or NVIDIA. This highlights the growing importance of GPU acceleration in modern HPC systems, where massive parallel processing capabilities enable substantially higher computational performance and greater energy efficiency than CPU-only systems for many scientific and engineering applications. The widespread adoption of GPUs in the TOP500 rankings demonstrates that heterogeneous, accelerator-based computing has become the dominant architecture for today’s most powerful supercomputers.
Practical Challenges in GPU Computing¶
Despite their advantages, GPUs are not a universal replacement for CPUs. Their benefits are most significant for highly parallel workloads. Applications with limited parallelism, complex control flow, or frequent CPU-GPU communication may achieve little or no performance improvement. Therefore, choosing between CPU and GPU computing requires careful consideration of the application’s characteristics.
Limited applicability: Not all workloads can benefit from GPU acceleration. Applications with irregular memory access patterns, extensive branching, or limited parallelism may see little performance improvement.
Programming complexity: GPU programming requires specialized knowledge of parallel algorithms, GPU architectures, and programming frameworks, resulting in a steeper learning curve compared with traditional CPU programming.
Data transfer overhead: Communication between the CPU and GPU can become a performance bottleneck due to limited interconnect bandwidth. Minimizing data movement is often essential for achieving high GPU performance.
Higher development effort: Existing CPU-based applications may require significant code modifications or redesign to effectively utilize GPU resources.
Keypoints
GPUs are specialized accelerators designed for specific types of computational workloads.
Highly parallel and compute-intensive applications are particularly well suited for GPU acceleration.
GPU-accelerated systems dominate the highest ranks of the TOP500 supercomputer list.
Effective GPU utilization requires new programming skills and knowledge of parallel computing concepts.