Heterogeneity
Interconnection
Topologies
Parallel and Distributed Computing
Arfan Shahzad
{ [email protected] }
Introduction to Heterogeneity
• Heterogeneous computing refers to systems that incorporate multiple
types of computing resources, such as CPUs, GPUs, FPGAs (Field-
Programmable Gate Arrays), TPUs (Tensor Processing Units), and
other specialized accelerators, to execute computational workloads
efficiently.
Introduction to Heterogeneity cont…
• Modern computing systems are increasingly heterogeneous because
different types of processors are optimized for different tasks. For example:
• CPUs handle general-purpose computing tasks efficiently.
• GPUs excel in parallel processing for graphics and deep learning.
• FPGAs are configurable and provide high-performance computation for specific
applications.
• TPUs are designed for accelerating deep learning inference and training.
Introduction to Heterogeneity cont…
Types of Heterogeneous Computing
1. Hardware Heterogeneity: Involves different types of hardware architectures
such as CPUs, GPUs, FPGAs, and TPUs.
2. Software Heterogeneity: Uses different programming models, languages, and
frameworks to utilize heterogeneous hardware (e.g., CUDA, OpenCL, SYCL, and
TensorFlow).
3. System-Level Heterogeneity: Integrates multiple computing devices within a
distributed system or cloud computing environment.
Introduction to Heterogeneity cont…
Heterogeneous Computing Architectures
1. CPU-GPU Systems: CPUs handle sequential operations, while GPUs process parallel workloads
(e.g., CUDA, OpenCL).
2. Heterogeneous Multi-Core Systems: Combinations of different cores (big.LITTLE architecture,
ARM architectures).
3. Reconfigurable Computing Systems: Use FPGAs to adapt to different workloads dynamically.
4. Cloud-Based Heterogeneous Systems: Distributed computing using multiple types of
virtualized hardware in the cloud.
Introduction to Heterogeneity cont…
Challenges in Heterogeneous Computing
1. Programming Complexity: Requires different programming models (CUDA for
GPUs, VHDL for FPGAs, etc.).
2. Load Balancing: Efficiently distributing tasks across different processing units.
3. Data Transfer Bottlenecks: Moving data between heterogeneous components
(e.g., CPU to GPU memory).
4. Energy Efficiency – Optimizing power consumption while utilizing multiple
qcomputing units.
Interconnection Topologies
• Interconnection topology refers to the structure and organization of
communication links between processing elements (PEs) in a parallel
or distributed computing system.
• The choice of topology affects performance, scalability, latency, and
fault tolerance.
Interconnection Topologies cont…
Classification of Interconnection Networks
• Direct Interconnection Networks: Processing elements are connected
directly to each other (e.g., Mesh, Hypercube).
• Indirect Interconnection Networks: Use switching elements for
communication (e.g., Crossbar, Multistage networks).
Interconnection Topologies cont…
Common Interconnection Topologies
Topology Description Advantages Disadvantages
Single communication channel Simple, cost-effective, Bandwidth limitation, contention,
Bus
shared among processors. easy to implement. single point of failure.
Processors connected in a No central point of High latency, network failure
Ring
circular fashion. failure, simple routing. disconnects nodes.
Mesh Nodes connected in a grid-like Scalability, predictable High diameter (more hops needed
(2D/3D) structure. communication paths. for communication).
Each node connects to log2(N) Low diameter, efficient
Hypercube High complexity in large systems.
other nodes. communication.
Hierarchical connection of Good scalability, Single point of failure at higher
Tree
nodes. efficient data levels.
Each processor has a dedicated High-speed, parallel
Crossbar Expensive, complex wiring.
link to each memory module. access.
Multistage Uses multiple switching stages High performance, Higher cost, complex routing
Networks for routing. scalable. algorithms.
Interconnection Topologies cont…
Key Topologies: Bus Topology
• All nodes share a common communication channel.
• Suitable for small-scale systems.
• Used in early multiprocessor systems.
• Example: Traditional motherboard communication.
Interconnection Topologies cont…
Key Topologies: Ring Topology
• Each processor connects to two neighbors, forming a closed loop.
• Data circulates in one or both directions.
• Used in: Token Ring Networks, Optical Networks.
Interconnection Topologies cont…
Key Topologies: Mesh Topology (2D and 3D)
• Nodes are arranged in a grid structure with direct connections to
neighbors.
• 2D Mesh is widely used in Network-on-Chip (NoC) architectures.
• Example: IBM Blue Gene Supercomputer.
Interconnection Topologies cont…
Key Topologies: Hypercube Topology
• A generalization of a cube where each node connects to log2(N)
neighbors.
• Efficient for large-scale parallel processing.
• Used in: Cray T3D, Intel Paragon.
Interconnection Topologies cont…
Key Topologies: Tree Topology
• Hierarchical structure with nodes connected in parent-child
relationships.
• Used in interprocessor communication and parallel databases.
• Example: Fat Tree architecture in data centers.
Interconnection Topologies cont…
Key Topologies: Crossbar Topology
• A dedicated switch connects each processor-memory pair.
• High-speed, but expensive and difficult to scale.
• Used in: High-performance computing (HPC) and mainframe
computers.
Interconnection Topologies cont…
Key Topologies: Multistage Interconnection Networks (MINs)
• Combination of switching elements to create multiple communication
stages.
• Used in large supercomputers and data centers.
• Example: Omega, Butterfly, and Banyan networks.
Interconnection Topologies cont…
Interconnection Topologies for Scalability & Performance
Topology Scalability Latency Fault Tolerance Cost
Bus Low High Low Low
Ring Medium Medium Medium Low
Mesh High Medium High Medium
Hypercube High Low High High
Tree High Low Medium Medium
Crossbar Low Very Low High Very High
Multistage High Low High High
Interconnection Topologies cont…
Interconnection Networks in Modern Systems
• Data Centers & Cloud Computing: Use high-speed Fat Tree, Torus, and
Dragonfly networks.
• HPC & Supercomputers: Hypercube, 3D Mesh, and Crossbar architectures
improve communication speed.
• Network-on-Chip (NoC): Used in multi-core processors (e.g., Intel’s Tile-
based architecture).
