Upload
sheryl-gibson
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
1
Parallel Scientific Computing: Algorithms and Tools
Lecture #3
APMA 2821A, Spring 2008
Instructors: George Em Karniadakis
Leopold Grinberg
2
Levels of ParallelismJob level parallelism: Capacity computing
Goal: run as many jobs as possible on a system for given time period. Concerned about throughput; Individual user’s jobs may not run faster.
Of interest to administratorsProgram/Task level parallelism: Capability
computingUse multiple processors to solve a single problem.Controlled by users.
Instruction level parallelism:Pipeline, multiple functional units, multiple cores.Invisible to users.
Bit-level parallelism:Of concern to hardware designers of arithmetic-logic
units
3
Granularity of Parallel Tasks
Large/coarse grain parallelism:Amount of operations that run in parallel is
fairly largee.g., on the order of an entire program
Small/fine grain parallelism:Amount of operations that run in parallel is
relatively smalle.g., on the order of single loop.
Coarse/large grains usually result in more favorable parallel performance
4
Flynn’s Taxonomy of Computers
SISD: Single instruction stream, single data stream
MISD: Multiple instruction streams, single data stream
SIMD: Single instruction stream, multiple data streams
MIMD: Multiple instruction streams, multiple data streams
5
Classification of Computers
SISD: single instruction single dataConventional computersCPU fetches from one instruction stream and
works on one data stream.Instructions may run in parallel (superscalar).
MISD: multiple instruction single dataNo real world implementation.
6
Classification of ComputersSIMD: single instruction multiple data
Controller + processing elements (PE)Controller dispatches an instruction to PEs; All PEs
execute same instruction, but on different datae.g., MasPar MP-1, Thinking machines CM-1, vector
computers (?)MIMD: multiple instruction multiple data
Processors execute own instructions on different data streams
Processors communicate with one another directly, or through shared memory.
Usual parallel computers, clusters of workstations
7
Flynn’s Taxonomy
8
Programming Model
SPMD: Single program multiple data
MPMD: multiple programs multiple data
9
Programming Model
SPMD: Single program multiple dataUsual parallel programming modelAll processors execute same program, on
multiple data sets (domain decomposition)Processor knows its own ID
•if(my_cpu_id == N){}•else {}
10
Programming Model
MPMD: Multiple programs multiple dataDifferent processors execute different
programs, on different dataUsually a master-slave model is used.
• Master CPU spawns and dispatches computations to slave CPUs running a different program.
Can be converted into SPMD model•if(my_cpu_id==0) run function_containing_program_1;
•else run function_containing_program_2;
11
Classification of Parallel Computers
Flynn’s MIMD computers contain a wide variety of parallel computers
Based on memory organization (address space):Shared-memory parallel computers
• Processors can access all memoriesDistributed-memory parallel computers
• Processor can only access local memory• Remote memory access through explicit
communication
12
Shared-Memory Parallel Computer
Superscalar processors with L2 cache connected to memory modules through a bus or crossbar All processors have access to all
machine resources including memory and I/O devices
SMP (symmetric multiprocessor): if processors are all the same and have equal access to machine resources, i.e. it is symmetric.
SMP are UMA (Uniform Memory Access) machines
e.g., A node of IBM SP machine; SUN Ultraenterprise 10000
Prototype shared-memory parallel computer
P – processor; C – cache; M – memory.
Bus or crossbar
M1
P1
C
M2
P2
C
M3
P3
C
Mn
Pn
C
…
…
memory
13
Shared-Memory Parallel Computer If bus,
Only one processor can access the memory at a time.
Processors contend for bus to access memory
If crossbar, Multiple processors can access
memory through independent paths
Contention when different processors access same memory module
Crossbar can be very expensive. Processor count limited by
memory contention and bandwidth Max usually 64 or 128
…
…
P1
C
M1 M2 Mn
P2
C
Pn
C
bus
M1
P1
C
M2
P2
C
M3
P3
C
Mn
Pn
C
…
…
memory
crossbar
memory
14
Shared-Memory Parallel Computer
Data flows from memory to cache, to processors
Performance depends dramatically on reuse of data in cacheFetching data from memory with potential
memory contention can be expensiveL2 cache plays of the role of local fast
memory; Shared memory is analogous to extended memory accessed in blocks
15
Cache CoherencyIf a piece of data in one processor’s cache
is modified, then all other processors’ cache that contain that data must be updated.
Cache coherency: the state that is achieved by maintaining consistent values of same data in all processors’ caches.
Usually hardware maintains cache coherency; System software can also do this, but more difficult.
16
Programming Shared-Memory Parallel Computers
All memory modules have the same global address space.Closest to single-processor computerRelatively easy to program.
Multi-threaded programming:Auto-parallelizing compilers can extract fine-grain
(loop-level) parallelism automatically;Or use OpenMP;Or use explicit POSIX (portable operating system
interface) threads or other thread libraries.Message passing:
MPI (Message Passing Interface).
17
Distributed-Memory Parallel Computer
Superscalar processors with local memory connected through communication network.
Each processor can only work on data in local memory
Access to remote memory requires explicit communication.
Present-day large supercomputers are all some sort of distributed-memory machines
Communication Network
P1
M
P2
M
Pn
M…
Prototype distributed-memory computer
e.g. IBM SP, BlueGene; Cray XT3/XT4
18
Distributed-Memory Parallel Computer
High scalabilityNo memory contention such as those in
shared-memory machinesNow scaled to > 100,000 processors.
Performance of network connection crucial to performance of applications.Ideal: low latency, high bandwidth
Communication much slower than local memory read/writeData locality is important. Frequently used data local memory
19
Programming Distributed-Memory Parallel Computer
“Owner computes” rule Problem needs to be broken up into independent tasks with
independent memory Each task assigned to a processor Naturally matches data based decomposition such as a domain
decomposition Message passing: tasks explicitly exchange data by
message passing. Transfers all data using explicit send/receive instructions User must optimize communications Usually MPI (used to be PVM), portable, high performance
Parallelization mostly at large granularity level controlled by user Difficult for compilers/auto-parallelization tools
20
Programming Distributed-Memory Parallel Computer
A global address space is provided on some distributed-memory machine Memory physically distributed, but globally addressable; can be
treated as “shared-memory” machine; so-called distributed shared-memory.
Cray T3E; SGI Altix, Origin. Multi-threaded programs (OpenMP, POSIX threads) can also be
used on such machines User accesses remote memory as if it were local; OS/compilers
translate such accesses to fetch/store over the communication network.
But difficult to control data locality; performance may suffer. NUMA (non-uniform memory access); ccNUMA (cache coherent
non-uniform memory access); overhead
21
Hybrid Parallel Computer
Overall distributed memory, SMP nodes
Most modern supercomputers and workstation clusters are of this type
Message passing; or hybrid message passing/threading.
M M
Bus or crossbar
P P
M M
Bus or crossbar
P P
Communication network
……
Hybrid parallel computer
e.g. IBM SP, Cray XT3
22
Interconnection Network/Topology
Nodes, links Neighbors: nodes with a link between them Degree of a node: number of neighbors it has Scalability: increase in complexity when more nodes are added.
Ring Fully connected network
23
Topology
Hypercube
24
Topology
1D/2D mesh/torus
3D mesh/torus
25
Topology
Tree Star
26
Topology
Bisection width: minimum number of links that must be cut in order to divide the topology into two independent networks of the same size (plus/minus one node)
Bisection bandwidth: communication bandwidth across the links that are cut in defining bisection width
Larger bisection bandwidth better