Parallel SystemsWelcome to cs380p

Chris Rossbach + Calvin Lin

CS380p

Introduction 1CS380P

Outline for Today

• Course Overview

• Course Details and Logistics

• Concurrency & Parallelism Basics• Motivation• Problem Decomposition

Acknowledgments: some materials in this lecture borrowed from or built on materials from:

• Emmett Witchel, who borrowed them from: Kathryn McKinley, Ron Rockhold, Tom Anderson, John Carter, Mike Dahlin, Jim Kurose, Hank Levy, Harrick Vin, Thomas Narten, and Emery Berger

• Mark Silberstein, who borrowed them from: Blaise Barney, Kunle Olukoton, Gupta

Introduction 2CS380P

Course Details

Course Name: CS380P – Parallel Systems

Lectures: Online

Class Web Page: http://www.cs.utexas.edu/users/rossbach/cs380p

Instructors: Chris Rossbach + Calvin Lin

Text: Principles of Parallel Programming (ISBN-10: 0321487907)

Please read the syllabus!

Introduction 3CS380P

http://www.cs.utexas.edu/users/rossbach/cs378hhttp://www.cs.utexas.edu/users/rossbachhttp://www.cs.utexas.edu/users/lin/https://www.universitycoop.com/search?keywords=Lin%20Snyder%20Parallel%20Programming

Why you should take this course

Introduction 4CS380P

Why you should take this course

• Parallelism is super-cool and super-important

Introduction 4CS380P

Why you should take this course

• Parallelism is super-cool and super-important

• You’ll learn important concepts and background

Introduction 4CS380P

Why you should take this course

• Parallelism is super-cool and super-important

• You’ll learn important concepts and background

• Have fun programming cool systems• GPUs! Multi-core!

• Modern infrastructure and programming languages

• Interesting synchronization primitives (not just about locks!)

Introduction 4CS380P

Why you should take this course

• Parallelism is super-cool and super-important

• You’ll learn important concepts and background

• Have fun programming cool systems• GPUs! Multi-core!

• Modern infrastructure and programming languages

• Interesting synchronization primitives (not just about locks!)

Two perspectives:• The “just eat your kale and quinoa” argument• The “it’s going to be fun” argument

Introduction 4CS380P

My first computer

Introduction 5CS380P

My first computer

Introduction 5CS380P

My first computer

CPU

Introduction 5CS380P

My first computer

CPU

Storage

Introduction 5CS380P

My first computer

CPU

Storage

Tape drive!(also good for playing heavy metal music)

Introduction 5CS380P

My first computer

CPU

Storage

screen

Tape drive!(also good for playing heavy metal music)

Introduction 5CS380P

My first computer

Wires + gobbledygook

CPU

Storage

screen

Tape drive!(also good for playing heavy metal music)

Introduction 5CS380P

My current computer

Introduction 6CS380P

My current computer

Too boring…

Introduction 6CS380P

Another of my current computers

Introduction 7CS380P

Another of my current computers

Introduction 7CS380P

Another of my current computers

GPU

Image DSP

Crypto

CPU

…

CPU

Introduction 7CS380P

Another of my current computers

GPU

Image DSP

Crypto

CPU

…

CPU

Introduction 7CS380P

Another of my current computers

GPU

Image DSP

Crypto

CPU

…

CPU

A lot has changed but…the common theme is…??

Introduction 7CS380P

Modern Technology Stack

Introduction 8CS380P

Modern Technology Stack

Introduction

CPU I/O dev DISK NIC

HW

8CS380P

Modern Technology Stack

Introduction

CPU I/O dev DISK NIC

HW

8CS380P

Modern Technology Stack

Introduction

CPU I/O dev DISK NIC

ApplicationsApplications

HW

8CS380P

Modern Technology Stack

Introduction

driver driver driver

CPU I/O dev DISK NIC

process files pipes

LIBC/CLR

process files pipes

ApplicationsApplications

user

kernel

OS-levelabstractions

HAL

user-modeRuntimes/libs

HW

8CS380P

Modern Technology Stack

Introduction

GPU

HW FPGA ASIC

DSPNVM CRYPT

8CS380P

Modern Technology Stack

Introduction

Vendor-specific

driver

GPU

ioctl

Runtime

deviceAPIs

ApplicationsApplications

user

kernel

HW

Runtimesupportmmap

FPGA ASIC

DSPNVM CRYPT

Runtime

deviceAPIs

Runtime

deviceAPIs

Runtime

deviceAPIs

8CS380P

Concurrency and Parallelism are Everywhere

Introduction

CPU DISKGPU

FPGA

ASIC

DSPNVM CRYPT

driver driver driver

ApplicationsApplications

Vendor-specific

driver

ioctl

Runtime

deviceAPIs

ApplicationsApplications

mmap

Runtime

deviceAPIs

Runtime

deviceAPIs

Runtime

deviceAPIs

9CS380P

HYPERVISORHYPERVISOR

vCPU vDISKvGPU

vFPGA

vASIC

vDSPvNVM vCRYPT

Concurrency and Parallelism are Everywhere

Introduction

CPU DISKGPU

FPGA

ASIC

DSPNVM CRYPT

driver driver driver

ApplicationsApplications

Vendor-specific

driver

ioctl

Runtime

deviceAPIs

ApplicationsApplications

mmap

Runtime

deviceAPIs

Runtime

deviceAPIs

Runtime

deviceAPIs

9CS380P

ApplicationsApplications

driver driver driver

vCPU vDISK

ApplicationsApplications

Vendor-specific

d r i v e r

vGPU

ioctl

Runtime

deviceAPIs

mmap

vFPGA

vASIC

vDSPvNVM vCRPT

Runtime

deviceAPIs

Runtime

deviceAPIs

Runtime

deviceAPIs

CPU DISKGPU

FPGA

ASIC

DSPNVM CRYPT

HYPERVISORHYPERVISOR

Introduction

Concurrency and Parallelism are Everywhere

10CS380P

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are Everywhere

10CS380P

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Cluster OSCluster OS

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

ApplicationsApplications

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Cluster OSCluster OS

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Introduction

ApplicationsApplications

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Cluster OSCluster OS

Concurrency and Parallelism are Everywhere

10CS380P

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

driver driver driver

vCPUvDIS

K

Vendor-specific

d r i v e rvGP

U

ioctl

Runtime

deviceAPI

s

mmap

vFPGA

vASIC

vDSP

vNVM

vCRPT

Runtime

deviceAPI

s

Runtime

deviceAPI

s

Runtime

deviceAPI

s

CPU DISKGPU

FPGA

ASIC

DSPNVMCRYPT

HYPERVISORHYPERVISOR

Concurrency and Parallelism are everywhere

GPU

Image DSP

Crypto

CPU(s)

…

GPUCrypto

FPGADSP

Introduction 11CS380P

Concurrency and Parallelism are everywhere

GPU

Image DSP

Crypto

CPU(s)

…

GPUCrypto

FPGADSP

Introduction 11CS380P

Concurrency and Parallelism are everywhere

GPU

Image DSP

Crypto

CPU(s)

…

GPUCrypto

FPGADSP

Key concerns:

Introduction 11CS380P

Concurrency and Parallelism are everywhere

GPU

Image DSP

Crypto

CPU(s)

…

GPUCrypto

FPGADSP

Key concerns:• Concurrency/parallelism can’t be avoided

anymore (want a job?)• A program or two playing with locks and threads

isn’t enough

Course goal is to expose you to lots of ways of programming systems like these

Introduction 11CS380P

Goal: Make Parallelism Your Close FriendMethod: Use Many Different Approaches

Abstract Concrete

Locks and Shared Memory Synchronization Basic Locking

Prefix sum – pthreads

Language SupportGo lab: condition variables, channels, go routines

Rust lab: type-safety, 2PC

Parallel Architectures GPU Programming Lab

HPC MPI: Barnes-Hut lab

Modern/Advanced Topics

• Specialized Runtimes / Programming Models

• Auto-parallelization

• Race Detection

Introduction 12CS380P

Serial vs. Parallel Program

Introduction 13CS380P

Serial vs. Parallel Program

Introduction 13CS380P

Serial vs. Parallel Program

Introduction 13CS380P

Serial vs. Parallel ProgramKey concerns:

Introduction 13CS380P

Serial vs. Parallel ProgramKey concerns:• Programming model

Introduction 13CS380P

Serial vs. Parallel ProgramKey concerns:• Programming model• Execution Model

Introduction 13CS380P

Serial vs. Parallel ProgramKey concerns:• Programming model• Execution Model• Performance/Efficiency

Introduction 13CS380P

Serial vs. Parallel ProgramKey concerns:• Programming model• Execution Model• Performance/Efficiency• Exposing parallelism

Introduction 13CS380P

Technology Trends

Introduction 14CS380P

Introduction 15CS380P

Execution Models: Flynn’s Taxonomy

Introduction 16CS380P

Execution Models: Flynn’s Taxonomy

Introduction 16CS380P

Execution Models: Flynn’s Taxonomy

Normal Serial program

Introduction 16CS380P

Execution Models: Flynn’s Taxonomy

Uncommon architecture:Fault – tolerancePipeline parallelism Introduction 16CS380P

Execution Models: Flynn’s TaxonomyOur main focus

Introduction 16CS380P

SIMD

Introduction 17CS380P

SIMD

Introduction 17CS380P

• Example: vector operations (e.g., Intel SSE/AVX, GPU)

SIMD

Introduction 17CS380P

• Example: vector operations (e.g., Intel SSE/AVX, GPU)

SIMD

Introduction 17CS380P

MIMD

Introduction 18CS380P

MIMD

• Example: multi-core CPU

Introduction 18CS380P

MIMD

• Example: multi-core CPU

Introduction 18CS380P

Problem Partitioning

Introduction 19CS380P

Problem Partitioning

• Decomposition: Domain v. Functional

Introduction 19CS380P

Problem Partitioning

• Decomposition: Domain v. Functional

• Domain Decomposition (Data Parallel)• SPMD

• Input domain

• Output Domain

• Both

Introduction 19CS380P

Problem Partitioning

• Decomposition: Domain v. Functional

• Domain Decomposition (Data Parallel)• SPMD

• Input domain

• Output Domain

• Both

Introduction 19CS380P

Problem Partitioning

• Decomposition: Domain v. Functional

• Domain Decomposition (Data Parallel)• SPMD

• Input domain

• Output Domain

• Both

• Functional Decomposition (Task Parallel)• MPMD

• Independent Tasks

• Pipelining

Introduction 19CS380P

Problem Partitioning

• Decomposition: Domain v. Functional

• Domain Decomposition (Data Parallel)• SPMD

• Input domain

• Output Domain

• Both

• Functional Decomposition (Task Parallel)• MPMD

• Independent Tasks

• Pipelining

Introduction 19CS380P

Game of Life

Introduction 20CS380P

Game of Life

• Given a 2D Grid:

Introduction 20CS380P

Game of Life

• Given a 2D Grid:

What decomposition fits “best”?

• Domain (data parallel)

• Functional (task parallel)

Introduction 20CS380P

Domain decomposition

Introduction 21CS380P

Domain decomposition

Each CPU gets part of the input

Introduction 21CS380P

Domain decomposition

Each CPU gets part of the input

CPU 0

CPU 1

Introduction 21CS380P

Domain decomposition

Each CPU gets part of the input

CPU 0

CPU 1

For next time:

Introduction 21CS380P

Domain decomposition

Each CPU gets part of the input

CPU 0

CPU 1

For next time: • What issues/challenges might arise with this solution?

Introduction 21CS380P

Domain decomposition

Each CPU gets part of the input

CPU 0

CPU 1

For next time: • What issues/challenges might arise with this solution?• How could we do a functional decomposition?

Introduction 21CS380P