Parallel SystemsWelcome to cs380p
Chris Rossbach + Calvin Lin
CS380p
Introduction 1CS380P
Outline for Today
• Course Overview
• Course Details and Logistics
• Concurrency & Parallelism Basics• Motivation• Problem Decomposition
Acknowledgments: some materials in this lecture borrowed from or built on materials from:
• Emmett Witchel, who borrowed them from: Kathryn McKinley, Ron Rockhold, Tom Anderson, John Carter, Mike Dahlin, Jim Kurose, Hank Levy, Harrick Vin, Thomas Narten, and Emery Berger
• Mark Silberstein, who borrowed them from: Blaise Barney, Kunle Olukoton, Gupta
Introduction 2CS380P
Course Details
Course Name: CS380P – Parallel Systems
Lectures: Online
Class Web Page: http://www.cs.utexas.edu/users/rossbach/cs380p
Instructors: Chris Rossbach + Calvin Lin
Text: Principles of Parallel Programming (ISBN-10: 0321487907)
Please read the syllabus!
Introduction 3CS380P
http://www.cs.utexas.edu/users/rossbach/cs378hhttp://www.cs.utexas.edu/users/rossbachhttp://www.cs.utexas.edu/users/lin/https://www.universitycoop.com/search?keywords=Lin%20Snyder%20Parallel%20Programming
Why you should take this course
Introduction 4CS380P
Why you should take this course
• Parallelism is super-cool and super-important
Introduction 4CS380P
Why you should take this course
• Parallelism is super-cool and super-important
• You’ll learn important concepts and background
Introduction 4CS380P
Why you should take this course
• Parallelism is super-cool and super-important
• You’ll learn important concepts and background
• Have fun programming cool systems• GPUs! Multi-core!
• Modern infrastructure and programming languages
• Interesting synchronization primitives (not just about locks!)
Introduction 4CS380P
Why you should take this course
• Parallelism is super-cool and super-important
• You’ll learn important concepts and background
• Have fun programming cool systems• GPUs! Multi-core!
• Modern infrastructure and programming languages
• Interesting synchronization primitives (not just about locks!)
Two perspectives:• The “just eat your kale and quinoa” argument• The “it’s going to be fun” argument
Introduction 4CS380P
My first computer
Introduction 5CS380P
My first computer
Introduction 5CS380P
My first computer
CPU
Introduction 5CS380P
My first computer
CPU
Storage
Introduction 5CS380P
My first computer
CPU
Storage
Tape drive!(also good for playing heavy metal music)
Introduction 5CS380P
My first computer
CPU
Storage
screen
Tape drive!(also good for playing heavy metal music)
Introduction 5CS380P
My first computer
Wires + gobbledygook
CPU
Storage
screen
Tape drive!(also good for playing heavy metal music)
Introduction 5CS380P
My current computer
Introduction 6CS380P
My current computer
Too boring…
Introduction 6CS380P
Another of my current computers
Introduction 7CS380P
Another of my current computers
Introduction 7CS380P
Another of my current computers
GPU
Image DSP
Crypto
CPU
…
CPU
Introduction 7CS380P
Another of my current computers
GPU
Image DSP
Crypto
CPU
…
CPU
Introduction 7CS380P
Another of my current computers
GPU
Image DSP
Crypto
CPU
…
CPU
A lot has changed but…the common theme is…??
Introduction 7CS380P
Modern Technology Stack
Introduction 8CS380P
Modern Technology Stack
Introduction
CPU I/O dev DISK NIC
HW
8CS380P
Modern Technology Stack
Introduction
CPU I/O dev DISK NIC
HW
8CS380P
Modern Technology Stack
Introduction
CPU I/O dev DISK NIC
ApplicationsApplications
HW
8CS380P
Modern Technology Stack
Introduction
driver driver driver
CPU I/O dev DISK NIC
process files pipes
LIBC/CLR
process files pipes
ApplicationsApplications
user
kernel
OS-levelabstractions
HAL
user-modeRuntimes/libs
HW
8CS380P
Modern Technology Stack
Introduction
GPU
HW FPGA ASIC
DSPNVM CRYPT
8CS380P
Modern Technology Stack
Introduction
Vendor-specific
driver
GPU
ioctl
Runtime
deviceAPIs
ApplicationsApplications
user
kernel
HW
Runtimesupportmmap
FPGA ASIC
DSPNVM CRYPT
Runtime
deviceAPIs
Runtime
deviceAPIs
Runtime
deviceAPIs
8CS380P
Concurrency and Parallelism are Everywhere
Introduction
CPU DISKGPU
FPGA
ASIC
DSPNVM CRYPT
driver driver driver
ApplicationsApplications
Vendor-specific
driver
ioctl
Runtime
deviceAPIs
ApplicationsApplications
mmap
Runtime
deviceAPIs
Runtime
deviceAPIs
Runtime
deviceAPIs
9CS380P
HYPERVISORHYPERVISOR
vCPU vDISKvGPU
vFPGA
vASIC
vDSPvNVM vCRYPT
Concurrency and Parallelism are Everywhere
Introduction
CPU DISKGPU
FPGA
ASIC
DSPNVM CRYPT
driver driver driver
ApplicationsApplications
Vendor-specific
driver
ioctl
Runtime
deviceAPIs
ApplicationsApplications
mmap
Runtime
deviceAPIs
Runtime
deviceAPIs
Runtime
deviceAPIs
9CS380P
ApplicationsApplications
driver driver driver
vCPU vDISK
ApplicationsApplications
Vendor-specific
d r i v e r
vGPU
ioctl
Runtime
deviceAPIs
mmap
vFPGA
vASIC
vDSPvNVM vCRPT
Runtime
deviceAPIs
Runtime
deviceAPIs
Runtime
deviceAPIs
CPU DISKGPU
FPGA
ASIC
DSPNVM CRYPT
HYPERVISORHYPERVISOR
Introduction
Concurrency and Parallelism are Everywhere
10CS380P
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are Everywhere
10CS380P
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Cluster OSCluster OS
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
ApplicationsApplications
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Cluster OSCluster OS
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Introduction
ApplicationsApplications
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Cluster OSCluster OS
Concurrency and Parallelism are Everywhere
10CS380P
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
driver driver driver
vCPUvDIS
K
Vendor-specific
d r i v e rvGP
U
ioctl
Runtime
deviceAPI
s
mmap
vFPGA
vASIC
vDSP
vNVM
vCRPT
Runtime
deviceAPI
s
Runtime
deviceAPI
s
Runtime
deviceAPI
s
CPU DISKGPU
FPGA
ASIC
DSPNVMCRYPT
HYPERVISORHYPERVISOR
Concurrency and Parallelism are everywhere
GPU
Image DSP
Crypto
CPU(s)
…
GPUCrypto
FPGADSP
Introduction 11CS380P
Concurrency and Parallelism are everywhere
GPU
Image DSP
Crypto
CPU(s)
…
GPUCrypto
FPGADSP
Introduction 11CS380P
Concurrency and Parallelism are everywhere
GPU
Image DSP
Crypto
CPU(s)
…
GPUCrypto
FPGADSP
Key concerns:
Introduction 11CS380P
Concurrency and Parallelism are everywhere
GPU
Image DSP
Crypto
CPU(s)
…
GPUCrypto
FPGADSP
Key concerns:• Concurrency/parallelism can’t be avoided
anymore (want a job?)• A program or two playing with locks and threads
isn’t enough
Course goal is to expose you to lots of ways of programming systems like these
Introduction 11CS380P
Goal: Make Parallelism Your Close FriendMethod: Use Many Different Approaches
Abstract Concrete
Locks and Shared Memory Synchronization Basic Locking
Prefix sum – pthreads
Language SupportGo lab: condition variables, channels, go routines
Rust lab: type-safety, 2PC
Parallel Architectures GPU Programming Lab
HPC MPI: Barnes-Hut lab
Modern/Advanced Topics
• Specialized Runtimes / Programming Models
• Auto-parallelization
• Race Detection
Introduction 12CS380P
Serial vs. Parallel Program
Introduction 13CS380P
Serial vs. Parallel Program
Introduction 13CS380P
Serial vs. Parallel Program
Introduction 13CS380P
Serial vs. Parallel ProgramKey concerns:
Introduction 13CS380P
Serial vs. Parallel ProgramKey concerns:• Programming model
Introduction 13CS380P
Serial vs. Parallel ProgramKey concerns:• Programming model• Execution Model
Introduction 13CS380P
Serial vs. Parallel ProgramKey concerns:• Programming model• Execution Model• Performance/Efficiency
Introduction 13CS380P
Serial vs. Parallel ProgramKey concerns:• Programming model• Execution Model• Performance/Efficiency• Exposing parallelism
Introduction 13CS380P
Technology Trends
Introduction 14CS380P
Introduction 15CS380P
Execution Models: Flynn’s Taxonomy
Introduction 16CS380P
Execution Models: Flynn’s Taxonomy
Introduction 16CS380P
Execution Models: Flynn’s Taxonomy
Normal Serial program
Introduction 16CS380P
Execution Models: Flynn’s Taxonomy
Uncommon architecture:Fault – tolerancePipeline parallelism Introduction 16CS380P
Execution Models: Flynn’s TaxonomyOur main focus
Introduction 16CS380P
SIMD
Introduction 17CS380P
SIMD
Introduction 17CS380P
• Example: vector operations (e.g., Intel SSE/AVX, GPU)
SIMD
Introduction 17CS380P
• Example: vector operations (e.g., Intel SSE/AVX, GPU)
SIMD
Introduction 17CS380P
MIMD
Introduction 18CS380P
MIMD
• Example: multi-core CPU
Introduction 18CS380P
MIMD
• Example: multi-core CPU
Introduction 18CS380P
Problem Partitioning
Introduction 19CS380P
Problem Partitioning
• Decomposition: Domain v. Functional
Introduction 19CS380P
Problem Partitioning
• Decomposition: Domain v. Functional
• Domain Decomposition (Data Parallel)• SPMD
• Input domain
• Output Domain
• Both
Introduction 19CS380P
Problem Partitioning
• Decomposition: Domain v. Functional
• Domain Decomposition (Data Parallel)• SPMD
• Input domain
• Output Domain
• Both
Introduction 19CS380P
Problem Partitioning
• Decomposition: Domain v. Functional
• Domain Decomposition (Data Parallel)• SPMD
• Input domain
• Output Domain
• Both
• Functional Decomposition (Task Parallel)• MPMD
• Independent Tasks
• Pipelining
Introduction 19CS380P
Problem Partitioning
• Decomposition: Domain v. Functional
• Domain Decomposition (Data Parallel)• SPMD
• Input domain
• Output Domain
• Both
• Functional Decomposition (Task Parallel)• MPMD
• Independent Tasks
• Pipelining
Introduction 19CS380P
Game of Life
Introduction 20CS380P
Game of Life
• Given a 2D Grid:
Introduction 20CS380P
Game of Life
• Given a 2D Grid:
What decomposition fits “best”?
• Domain (data parallel)
• Functional (task parallel)
Introduction 20CS380P
Domain decomposition
Introduction 21CS380P
Domain decomposition
Each CPU gets part of the input
Introduction 21CS380P
Domain decomposition
Each CPU gets part of the input
CPU 0
CPU 1
Introduction 21CS380P
Domain decomposition
Each CPU gets part of the input
CPU 0
CPU 1
For next time:
Introduction 21CS380P
Domain decomposition
Each CPU gets part of the input
CPU 0
CPU 1
For next time: • What issues/challenges might arise with this solution?
Introduction 21CS380P
Domain decomposition
Each CPU gets part of the input
CPU 0
CPU 1
For next time: • What issues/challenges might arise with this solution?• How could we do a functional decomposition?
Introduction 21CS380P