WildFireWildFire: A Scalable Path : A Scalable Path for SMPsfor SMPs

Erick Hagersten and Michael Erick Hagersten and Michael KosterKoster

Sun Microsystems Inc.Sun Microsystems Inc.

Presented by Terry Arnold IIPresented by Terry Arnold II

IntroductionIntroduction

What was the goal?What was the goal?How did they achieve it?How did they achieve it?CMRCMRHASHASCompetitive ComparisonsCompetitive ComparisonsResultsResultsQuestionsQuestions

The GoalThe Goal

In the past people have been In the past people have been skeptical about the ability of SMPs to skeptical about the ability of SMPs to continue to scale due to their continue to scale due to their bandwidth limitationsbandwidth limitations

The trend has been to switch to cc-The trend has been to switch to cc-NUMANUMA

To improve the scalability of SMP To improve the scalability of SMP technologiestechnologies

Cc-NUMA issuesCc-NUMA issues

Great scalability but have less than Great scalability but have less than optimal “access patterns”optimal “access patterns”

Require high software optimization Require high software optimization for capacity and conflict missesfor capacity and conflict misses

Non trivial scheduling, etc. (resource Non trivial scheduling, etc. (resource and memory management)and memory management)

How?How?

The answer is the same as the answer The answer is the same as the answer to all engineering problems, that is, to all engineering problems, that is, throwing new acronyms at the throwing new acronyms at the problemproblem

Coherent Memory Replication (CMR)Coherent Memory Replication (CMR)Hierarchical Affinity Scheduling (HAS)Hierarchical Affinity Scheduling (HAS)Both of these exploit locality as a Both of these exploit locality as a

means of increasing performance means of increasing performance (that is for OLTP workloads)(that is for OLTP workloads)

The OverviewThe Overview

The Acronyms: CMRThe Acronyms: CMR

S-COMA with fixed home locations for each S-COMA with fixed home locations for each addressaddress

Shadow physical pagesShadow physical pages Coherence at hardware level (64 byte)Coherence at hardware level (64 byte) Things start out cc-NUMA and changed Things start out cc-NUMA and changed

into CMR based on hardware counters that into CMR based on hardware counters that monitor memory access patternsmonitor memory access patterns

Limitations – memory-resident pages and Limitations – memory-resident pages and large physical pages can only be large physical pages can only be replicated explicitlyreplicated explicitly

The Acronyms: HASThe Acronyms: HAS

Schedules in the following way:Schedules in the following way:Last processor it ran onLast processor it ran onSame node processorSame node processorRemote node processor (when load Remote node processor (when load

balances exceeds “threshold”)balances exceeds “threshold”)

ImplementationImplementation

2 ASICs – NIAC (coherence), NIDC (bit 2 ASICs – NIAC (coherence), NIDC (bit sliced interconnect)sliced interconnect)

These improve upon latency of a These improve upon latency of a switchswitch

NIAC – Interface and Global-NIAC – Interface and Global-Coherence LayerCoherence Layer

Translators and CountersTranslators and Counters

CompetitionCompetition The SGI Origin and The SGI Origin and

Sequent’s NUMA-QSequent’s NUMA-Q

Results 1Results 1

Results 2Results 2

Questions?Questions?

Is this “solution” too dependent on the software (kernel modifications)?

How compatible are CMR and HAS with the other DSM solutions?