Parallelism is Everywhere – So How Do We Make It Accessible? Daniel Ernst The University of Wisconsin-Eau Claire Parallelism is Everywhere – So How Do

Parallelism is Everywhere –So How Do We Make It Accessible?

Daniel ErnstThe University of Wisconsin-Eau Claire


Daniel Ernst

Department of Computer ScienceThe University of Wisconsin – Eau Claire



A Little Background …

• “Grew Up” (Graduate School) in Computer Architecture– Efficient, fast, low-power processors (2 ISCA, 1 MICRO papers, +)

• “Dabbled” in supercomputing– (Becoming a bigger part of my job now…)

• Became a CS educator

• Now– Trying to make sense of how these are all coming together



Moore’s Dividend



Where Did All The Performance Go?Software Development, c. 1950-2005

Figure byJim Larus[CACM 09]

“What Andy giveth, Bill taketh away.”



Can Multi-Core Supplant Moore’s “Dividend”?

• Double number of cores instead of speed

• NO, at least not without major innovation– Sequential code base will not scale without lots of help– Lack of parallel algorithms (and/or poor knowledge of them)– Difficult (or at least different) programming challenges– Fewer abstractions currently available

(for individual programs)



Parallel Algorithms

“In the context of sequential algorithms, it is standard practice to design more complex algorithms that outperform simpler ones (for example, by implementing a balanced tree instead of a list). For non-blocking algorithms, however, implementing more complex data structures has been prohibitively difficult.”

[Herlihy, Luchangco, Moir, Scherer, PODC 2003](Discussing a concurrent red-black tree)



Sequential to Parallel – A History• Existing code is sequential

• We tried parallel compilers in the ‘80s-’90s– Can get good code for a few specific cases, but not generally

• We tried to get the hardware to extract parallelism in 90’s-00’s– Dynamic mechanisms cannot find more than 2–4x parallelism

• The issue is with how programmers fundamentally think about problems– Not language specific

• If we’re training programmers to think only in sequence, how well will they be prepared for dealing with explicit parallelism?



Fundamental Take-aways

• Explicit parallelism is important, and is getting more so– Being able to identify it will be an increasingly marketable skill

• Understanding the underlying architectures is also becoming more important– Necessary to have at least basic understanding to extract reasonable amounts of

parallelism– Hybrid machines will make this critical if you want good performance

• ESPECIALLY before the tools catch up (but still after)



Programming Futures• There is a lack of consensus on the parallel

programming model– Data parallelism (Vector)– Unstructured (thread) parallelism

• (Edward Lee – “The Problem with Threads”)• Speculative threading?

– Message passing

• Lots of ongoing research with not much resolved• Moral: Everyone in the CS field needs to be prepared

to learn about new parallel programming methods and tools.

http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf



State of the Game

Educators This Talk

Industry

This (slightly modified) graph shamelessly stolen from Jim Larus



An Opportunistic Approach

• Typically, he/she who develops functionality to use fully use the hardware capabilities “wins”– Google, facebook, intel, Microsoft, etc. all took advantage of technological

inflection points

• So, if we don’t know what to teach yet, how do we prepare our students to be leaders?



Some Context• UW – Eau Claire“Founded in 1916, UW-Eau Claire is one of the Midwest’s top public universities. UW-Eau

Claire provides students with small classes led by talented professors, cutting-edge programs, hands-on research and life-enhancing study abroad opportunities.

Motto: Excellence. Our measure, our motto, our goal.• What that actually means:

– Approx. 11k students, only 500 are grad students– Focus is undergraduate engagement

• Teaching excellence• Undergraduate research



UW – Eau Claire Computer Science• ABET Accredited comprehensive CS major• 6 faculty (+1 vacancy), 1 part-time academic staff• Approximately 40-50 graduates per year in 3 majors:

– Comprehensive CS– Software Engineering– Computer Engineering (CS degree with CE emphasis)

• We’re filthy rich (sort of)

• Constituencies / Industrial Partners / Landing Spots– Healthcare, insurance (Marshfield Clinic, NW Mutual)– Some systems development (Cray/IBM)



UW – Eau Claire Computer Science• Focus is on keeping an innovative curriculum

– A majority of our scholarship is within this scope– SIGCSE-related activity is large

• total of 31 papers and 7 panels presented in SIGCSE sponsored conferences and journals since 2002

– The department has recently been nominated three times as the University of Wisconsin Teaching Department of the Year

(2001, 2002, 2003)• Curricular Basics:

– Java early, C/C++ later, plus a full PL course– Full P&H Comp Org.– Late: strong software engineering coursework along with robust systems

classes (OS/Networks)Not for the casual student



Parallelism in the Curriculum – Current• Many (most?) schools treat parallel computing as an independent

topic, addressed in the senior year as an elective.



Curricular Goals at UW-ECStudents should gain:

– a generic understanding of computational concurrency and parallelism.• see parallelism in problems• be able to exploit it in a reasonable way

– exposure to parallelism as an orthogonal method with respect to other standard curricular items.• Don’t just teach parallelism by itself – integrate it in with other

elements as simply another tool in the programming box.• Teaching it in context can lower time overhead, as well

– confidence in using parallelism as a tool.



Principles for Implementation

• Simplicity– Keep to easy (deterministic) and natural concepts

• Ubiquity– More than a “1-shot” issue

• Accessibility– Don’t make the learning curve steeper than it has to be



Keeping Parallelism Simple• First, tweak the underlying concept of a computer

– A “collection of brains”

• The concepts behind basic parallelism are simple– Sharing work

• Different tasks• Split up data

• Especially if they’re given a more natural context

Simplicity



Natural Parallelism/Concurrency• We have very natural allies in the educational graphics/animation

toolsets– Alice, Scratch, etc.

• These tools make basic concurrency natural and largely deterministic– easier to debug!

Simplicity



Another Natural Example – Swarm Intelligence• “Creative” Monte Carlo solution to TSP

– Instantiate multiple agents to probabilistically discover the shortest trip– “Threads as Entities”

• Each agent (“ant”) operates in its own thread– ants leave pheromone trails along arcs to influence later ants

• Arc information needs to be synchronized

• Orthogonal topics:– Monte Carlo– heuristics

Simplicity



Other “Clean” Parallel ExamplesSimplicity



Parallelism Everywhere Parallelism the Norm• Unlike the past, sequential code written today is likely to

only see very small (if any) gains from hardware upgrades. – As the number of cores increases, non-parallel performance

will become relatively more unacceptable in practice.

• Exploiting parallelism needs to be a first-class citizen– (or at least second-class)– not an item studied once and then not used again.– Ubiquitous

Ubiquity



An Integrated Model at UW-EC• CS1 – Decomposition and Assignment

– Introduce methods for recognizing obvious parallelism– Practice with simple, natural, and integrated examples

• CS2 – Basic Orchestration– Continued practice with obvious parallelism– Introduce communication

• Algorithms – Parallelism Limits– Extend algorithm analysis to parallel algorithms– Introduce more complicated communication patterns

• Operating Systems – Synchronization Constructs– “The usual topics “

Ubiquity



After a Reality Check…• CS2 – Decomposition and Assignment

– Introduce methods for recognizing obvious parallelism– Practice with simple, natural, and integrated examples

• CS2/Algorithms – Basic Orchestration– Continued practice with obvious parallelism– Introduce communication

• Algorithms/OS – Parallelism Limits– Extend algorithm analysis to parallel algorithms– Introduce more complicated communication patterns

• Operating Systems – Synchronization Constructs– “The usual topics “

Ubiquity



CS2 Possibilities – Image Mosaic• “Nifty Assignment” (Pattis, SIGCSE 2005)

– create an image out of tiles ofsmaller images

– tiles are independent(unless you want it to look good!)

• Basic parallelism is easy, butgood quality needs synchronization– Payoff!

• Can vary synchronization difficulty

• Orthogonal topics:– searching/sorting/matching

Ubiquity



The Role of Architecture• Understanding how a program “lays out” on a machine.

– What is it capable of?– What are the bottlenecks?

• My take thus far:– Decreased emphasis on assembly programming– Slightly decreased emphasis on microarchitecture

• (Depending on program needs)– Increased emphasis on system-level view (interaction between processor

and memory)

Ubiquity



Keeping Parallelism Accessible• For each individual curriculum, the choice of language/platform

has a significant effect on accessibility.

• At UW-EC, we are largely a “Java shop”, with some C/C++– To keep learning curve gentle, we try to operate within that framework– Java.util.concurrent / OpenMP

• Taking the time to introduce a new platform eats far too much time and can lead to projection of learning difficulties*– (and they’re just going to change, anyways)

The concepts are more important than the platform

*YMMV

Accessibility



The Delicate Balance

While we want to have student learning to be as platform-agnostic as possible…

…they also need to learn real skills for the workforce…

…and they need to practice with the concepts somehow!

Accessibility



Current Challenges1. General Workload

– Especially given the ubiquity argument, there’s a lot of material that needs to be created/re-worked in new contexts

– How do we motivate instructors?

2. Faculty Readiness– At least anecdotally, the percentage of faculty who feel

comfortable teaching parallel computing is painfully small– How can we help with faculty accessibility?

3. Future Uncertainty– Given that we’re not quite sure what future programming

models will look like – what topics should we be teaching?



Current Challenges (More?!?)4. Availability of Tools/Language Support for Parallelism

– There are very few language/toolset combinations that collectively fulfill the major criteria for curricular languages:

• Language used in industry (or related to one)• Supports OO• Good support for “easy” parallelism• Not syntactic suicide



The Reality of Implementation

• If change is really going to be cross-curriculum, it requires that there is significant buy-in from every faculty/staff member that teaches any course that includes the new material.

• This is a serious professional development challenge



EAPF – Addressing The ChallengesEducational Alliance for a Parallel Future• Combined effort of Industry:

– Intel, nVidia, AMD, Microsoft, etc.• and Academia

– Members range from research (UIUC/Berkeley) to supercomputing experts to small college educators

• Goals:– Increase awareness– Encourage and support innovation in training and tools

• Keep an eye out for more…



Thank You!

Q & A

Documents

Parallelism is Everywhere – So How Do We Make It Accessible? Daniel Ernst The University of Wisconsin-Eau Claire Parallelism is Everywhere – So How Do