Performance and energy efficiency are determined less by programming language alone and more by how well software design aligns with the underlying hardware. While hardware choices are largely outside a programmer’s direct control, they strongly influence performance, making it essential to design software that works with - not against - them.
This is particularly important for large-scale parallel programming where bottlenecks in memory accesses or communication, which may be hidden at smaller scales, can dominate runtime. As we scale up to use large systems, features of the processors, accelerators and interconnect architecture increasingly shape performance.
As such, we need to design whilst thinking ahead: anticipating bottlenecks, considering scalability, and targeting the machines on which the software will run.
This guide outlines a practical three-step approach: first, define programme goals with identification of likely bottlenecks; then, consider the key features of the target machine(s); and finally, plan the implementation based on the first two steps. Done well, this approach delivers major performance gains and energy savings from the beginning of the software lifecycle [R1].
The benefits of hardware-conscious programming
Before outlining this process, it’s useful to understand why this approach is effective. Hardware-conscious programming from the very start directly improves performance (time-to-solution) and energy efficiency by reducing unnecessary computation, data movement and idle time - three primary sources of inefficiency in software execution.
There are also important indirect advantages pertaining to the awareness produced (and ideally, documented in/alongside the code!) in the process. Without understanding which key hardware features impact performance, extensive changes could be required to the algorithmic, data, I/O or parallelism approach when moving to a new machine or optimising [R1].
Developing this awareness reduces that risk and helps to future-proof your code as problem size or system complexity increases. You anticipate how a move to a different platform will influence the code and what changes should be made to adapt it.
Furthermore, the behaviour of the programme will be largely anticipated and stable; issues such as memory contention or communication overheads are known and reduced.
In these ways, hardware-conscious programming improves portability, scalability and predictability.
A three step approach
But how can one get started? We recommend following our three step approach.
[IMAGE HERE]
Step 1: What dominates runtime?
The first step aligns with standard design practices: sustainability aside (for now!), what solution will be achieved, problem solved, or output(s) generated? Using this, we can anticipate major bottlenecks based on where most time and computations will occur.
Begin with identification of whether the performance will be limited mostly by how fast the processor can perform calculations (compute bound), or how fast the data can be moved (memory bound).
For example, dense matrix-matrix multiplication, ubiquitous in scientific computing and machine learning, is inherently compute bound. Each result element involves many floating-point operations relative to the data movement, giving it high arithmetic intensity. Consequently performance is typically limited by compute speed rather than memory access.
In contrast, sparse matrix-vector multiplication, common in large-scale simulation, is usually memory bound. Irregular access patterns and scattered data dominate runtime, making memory bandwidth and latency the main bottlenecks.
Step 2: What hardware constraints matter?
With this information, you can begin to map the demands of your programme to the system’s capabilities and constraints. Not all features are equally important: the aim is to identify those most likely to influence performance based on the anticipated bottlenecks. Key aspects of the target hardware/system include:
- Compute architecture: quantities of CPUs, GPUs, TPUs, and overall node design;
- Memory system: RAM size, hierarchy, bandwidth, and latency;
- Storage and I/O: storage type (HDD, SSD, hierarchical), access (local, network, or cloud), filesystem (distributed or parallel), and read/write performance;
- Network characteristics: bandwidth, latency, and communication patterns between nodes.
Step 3: What design choices follow?
The final step is to use our findings to build an implementation plan that meets the programme goals whilst efficiently using the target hardware. A good strategy is to adopt a cycle of continually measuring, profiling, and optimising, to corroborate improved performance, adjusting the plan where necessary, while maintaining a clear initial design.
First, decide which tasks run on which processing units: use CPUs for general tasks and GPUs/TPUs for highly parallel workloads [R2].
For compute-bound workloads, focus on maximising utilisation of processing units through parallelism and vectorisation. Use algorithms and hardware-aware libraries (e.g. CUDA for GPUs, optimised math libraries for TPUs) that expose parallelism while avoiding idleness. Minimise synchronisation and data transfer between host and device, ensuring compute-heavy computations can be offloaded where appropriate. For CPUs, design to match the architecture by partitioning data to match NUMA regions and match work to the available cores, avoiding oversubscription. Leverage vector/SIMD instructions for compute-intensive loops if supported, by using aligned, vector-friendly data layouts and minimising branching.
For memory-bound workloads, the priority is reducing data movement and improving access efficiency. Structure data to maximise cache locality by keeping related data together and favouring contiguous memory over pointer-heavy data structures. Reuse buffers where possible to reduce allocation overhead. Since, performance is driven by data locality and access patterns [R3], these should be central considerations in programme design.
For storage and I/O, plan access patterns and checkpointing strategies from the outset [R4] . Reduce frequent small reads and writes via batch I/O operations, and pre-load large datasets into RAM to reduce repeated access to slower storage. Use compression where possible, and, for network-bound workloads, batch network requests and compress transmitted data to reduce bandwidth and energy usage.
Takeaway message / conclusion / next steps
Hardware cannot be neglected when considering the performance and sustainability of software. When successfully considered from the outset we can naturally embed portability, scalability and predictability.
This guide has outlined a three-step process for hardware-conscious programming: identify programme goals and anticipate bottlenecks; identify relevant hardware features; and finally bring those together to shape the programme design.
References
- Yantır, H. E., Eltawil, A. M., & Salama, K. N. (2022). A hardware/software co-design methodology for in-memory processors. Journal of Parallel and Distributed Computing, 161, 63-71.
- K, R., & Chiplunkar, N.N. (2018). A survey on techniques for cooperative CPU-GPU computing. Sustain. Comput. Informatics Syst., 19, 72-85.
- Usman, S., Mehmood, R., Katib, I., & Albeshri, A. (2023). Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and a Future System Architecture. Electronics, 12(1), 53. https://doi.org/10.3390/electronics12010053
- Kim, S., Sim, A., Wu, K. et al. Design and implementation of I/O performance prediction scheme on HPC systems through large-scale log analysis. J Big Data 10, 65 (2023). https://doi.org/10.1186/s40537-023-00741-4
Acknowledgements
This guide was written by Sadie Bartholomew and reviewed by Eleanor Broadway.
Find Sadie on:
Find Eleanor on: