Parallel Computing @ MSSteve TeixeiraPrincipal Product Unit ManagerParallel Developer ToolsMicrosoft Corporation

“Provide the platform, tools and broad ecosystem to reduce the complexity of HPC by making parallelism more accessible to address future computational needs.”

Microsoft’s Vision for HPC

Reduced Complexity Mainstream HPC Broad Ecosystem

Ease deployment forlarger scale clusters

Simplify management forclusters of all scale

Integrate with existing infrastructure

Address needs of traditional supercomputing

Address emerging cross-industry

computation trends

Enable non-technical users to harness the power of HPC

Increase number of parallel applications and codes

Offer choice of parallel development tools,

languages and libraries

Drive larger universe ofend-users, developers,

and system administrators

3

Looking Forward: Hard Problems

• Scaling distributed systems is hard• Data sets are increasing• Programming models are complex

4http://www.flickr.com/photos/maisonbisson/201844037/

We need simpler programming models

Crossing the Chasm• Embrace existing programming models

– We heart MPI• Increase reach of existing codes

– Cluster SOA, .NET/WCF, Excel integration• Invest in mainstream parallel dev tools

– Unlock multi/many-core for breadth developers– Evolve hybridized and scale-out models

• Seek opportunities for “automatic” parallelism– F#– DryadLINQ

Example: “Baby Names”

IEnumerable<BabyInfo> babies = ...;var results = new List<BabyInfo>();foreach (var baby in babies){ if (baby.Name == queryName && baby.State == queryState && baby.Year >= yearStart && baby.Year <= yearEnd) { results.Add(baby); }}results.Sort((b1, b2) => b1.Year.CompareTo(b2.Year));

Manual Parallel SolutionIEnumerable<BabyInfo> babies = …;var results = new List<BabyInfo>();int partitionsCount = Environment.ProcessorCount;int remainingCount = partitionsCount;var enumerator = babies.GetEnumerator();try { using (ManualResetEvent done = new ManualResetEvent(false)) { for (int i = 0; i < partitionsCount; i++) { ThreadPool.QueueUserWorkItem(delegate { var partialResults = new List<BabyInfo>(); while(true) { BabyInfo baby; lock (enumerator) { if (!enumerator.MoveNext()) break; baby = enumerator.Current; } if (baby.Name == queryName && baby.State == queryState && baby.Year >= yearStart && baby.Year <= yearEnd) { partialResults.Add(baby); } } lock (results) results.AddRange(partialResults); if (Interlocked.Decrement(ref remainingCount) == 0) done.Set(); }); } done.WaitOne(); results.Sort((b1, b2) => b1.Year.CompareTo(b2.Year)); }}finally { if (enumerator is IDisposable) ((IDisposable)enumerator).Dispose(); }

Synchronization Knowledge

Inefficient locking

Manual aggregation

Lack of foreach simplicity

Tricks

Heavy synchronization

Lack of thread reuse

Non-parallel sort

LINQ Solution

var results = from baby in babies where baby.Name == queryName && baby.State == queryState && baby.Year >= yearStart && baby.Year <= yearEnd orderby baby.Year ascending select baby;

.AsParallel()

Developer AccessibilityMake it easier to express and

manage the correctness, efficiency and maintainability

of parallelism on Microsoft platforms for developers of all

skill levels

Enable developers to express

parallelism easily and focus on the

problem to be solved

Improve the efficiency and

scalability of parallel applications

Simplify the process of designing and testing parallel

applications

Program to tasks, not threads Focus on problem domain C++, .NET, STM, agents, functional Embrace standards

Concurrency Runtime Enable latent parallelism Core OS changes (UMS) Scale-out runtimes (Dryad)

Debugger speaks prog models Debugger must scale out Correctness analysis Accessible profiling

• Microsoft HPC Web site– http://www.microsoft.com/hpc

• Parallelism at Microsoft– http://msdn.microsoft.com/concurrency

• Visual Studio 2010 Beta– http://msdn.microsoft.com/visualstudio

• Windows HPC TechCenter– http://technet.microsoft.com/en-us/hpc/default.aspx

• HPC on MSDN– http://code.msdn.microsoft.com/hpc

Additional Information

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of

the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Challenge: High Productivity Computing

“Make high-end computing easier and more productive to use.

Emphasis should be placed on time to solution, the major metric of value to high-end computing users…

A common software environment for scientific computation encompassing desktop to high-end systems will enhance productivity gains by promoting ease of use and manageability of systems.”

2004 High-End Computing Revitalization Task ForceOffice of Science and Technology Policy, Executive Office of the President

Systems Management

Job Scheduling

Networking& MPI Storage

New System Center UI PowerShell for CLI Management High Availability for Head Nodes Windows Deployment Services Diagnostics/Reporting Support for Operations Manager

Support for SOA and WCF Granular resource scheduling Improved scalability for larger

clusters New Job scheduling policies Interoperability via HPC Profile

NetworkDirect (RDMA) for MPI

Improved Network Configuration Wizard

Shared Memory MS-MPI for multi-core

MS-MPI integrated with Windows Event Tracing

Improved iSCSI SAN & parallel file system Support in Win2008

Improved Server Message Block ( SMB v2)

New 3rd party parallel file system support for Windows

New Memory Cache Vendors

What’s new in Windows HPC Server 2008?

15

HPC Server at scale• Data center integration

– HPC continues to integrate with the general computing fabric– Partnerships with System Center, MOAB, and more

• Commitment to performance– Open source contributions to ANL, OpenFabrics, and more– GP-GPU integration and tuning– Performance tuning labs– User mode and kernel mode RDMA

• Cloud integration– Offload or remote computation sometimes makes sense– Must solve problems with data locality, licensing, code installation

16

Approaching the storage problem• Segmentation models

– Structure models:

– Programming models:

Matrix Operations

• GEMM:C←αAB+βC

Pseudo Task Parallel

• foreach (file in dir)• parallel.foreach (…){

xxxx}

LINQ-like

• array{…}• map (func, array)

Size

• Large files• Lots of small files

Locality

• Server-read/server-write

• Server-read/local-write

• Schedule for local read (data locality)

Structure

• Unstructured (files)• Semi-structured

(HDF5, etc.)• Structured (SQL)

17

Insights• Many apps mine data without SQL

– Enormous amounts of data in unstructured flat files– Queries are too convoluted for SQL statement or SQL knowledge is limited– But… predictive analytics and general business intelligence are starting to use

HPC infrastructure

• No good general out of the box solutions– Pre-canned prog model and storage system works for some problems– Expensive PFS works well for existing and MPI apps– Some customers decide to build their own

Strategic issues• Our long term storage strategy is governed by three tensions:

– Success and timing of NFS v4.1– Stuff like Dryad/LINQ is loved by Emerging/Finance– Locality is critical

• Compute in the cloud makes the data locality problem especially complex

• General purpose vs. model specific– Model specific storage models are easier to build

• So many models to choose from…– MPI requires parallel file systems, not model based systems

Page 18

19

Models• We love MPI

– MPI cited as primary programming model for over 60% of users– We’re contributing to the MPI Forum– Open source contributions to MPICH at ANL are largest in MS history

• SOA & Horizontal applications– SOA innovations will help conveniently parallel applications scale to clusters– SOA infrastructure is designed to help horizontal apps scale– ISVs and developers can focus on writing code, not building middleware– Users focus on doing science rather than learning job control languages– Excel is used practically everywhere

• Accessible, “mass market” parallelism– How to leverage increasing core count on nodes and workstations– Parallel Computing developer technology shipping from MS soon– More to come: deeper integration of hybrid development models

Visual Studio 2010Tools/Programming Models/Runtimes

Parallel Pattern Library

Resource Manager

Task Scheduler

Task Parallel Library

Parallel LINQ

Managed NativeKey:

ThreadsOperating System

Concurrency Runtime

Programming Models

ThreadPool

Task Scheduler

Resource Manager

Data Structures D

ata

Stru

ctur

es

Tools

Tooling

ParallelDebugger

Tool Windows

Profiler Concurrency

Analysis

AgentsLibrary

UMS Threads

.NET Framework 4 Visual C++ 10Visual Studio

IDE

Windows