Field Programmable Gate Arrays: Power in Future HPC Systems Eric Stahlberg Ohio Supercomputer Center July 12, 2005

Field Programmable Gate Arrays:Field Programmable Gate Arrays:Power in Future HPC SystemsPower in Future HPC Systems

Eric StahlbergOhio Supercomputer Center

July 12, 2005

July 12, 2005 NCSA RSSI 2005

Discussion RoadmapDiscussion Roadmap

• Setting the context• Why FPGAs now?• Challenges to adoption• OpenFPGA as a bridge


Why Are People Here?Why Are People Here?

• FPGAs are cool• I want to learn valuable skills• I have a good idea I want to share• I have something to offer as a product or service• I want to learn about other developments in RC

• Where’s the free food?


About The Ohio Supercomputer CenterAbout The Ohio Supercomputer Center

• Located in Columbus, Ohio

• The Ohio Supercomputer Center was established in 1987 to position Ohio universities and industries at the forefront of computationally intensive research, development, engineering and networking.

• Delivering high performance computing and high capacity optical network for production and research

• Strategic aims OSC provides a reliable high-performance computing and communications infrastructure for a diverse,

statewide/regional community including education, academic research, industry, and state government.

OSC strives to be in the forefront of computational research in order to act as a key enabler for the state’s aspirations in advanced technology, information systems, and advanced industries, and,

OSC acts as a catalytic partner of Ohio universities and industries to enable

• Supporting High Performance Computing Requirements of- Bioinformatics- Advanced Manufacturing- Agriculture- Data-centric applications- Materials- Modeling, Testing and Instrumentation


Production Computing EnvironmentsProduction Computing Environments

SGI Altix – Linux IPF Shared Memory System

.20 TF64 GB32 900 MHz Intel Itanium 2

SUN COE with Time logic boards for specialized bionformatics apps

.04 TF48 GB48 900 MHz UltraSPARCIII

CRAY X1 AC .20 TF64 GB16 800 MHz vector processors

.23 TF

1.92 TF

.31 TF

2.5 TF

1.1 TF

Peak Performance

One of 8 sites around the country for computational grid testing and deployment

52 GB52 2.2 GHz Pentium 4 Xeon

Alliance Grid Testbed (AGT) Cluster

Cluster Ohio -- Distributed computational grid at 15 institutions around the state

736 GB192 550 MHz Pentium III Xeon, 144 733 MHz Itanium, 256 1.4 GHz AMD Athlons

BALE Visualization cluster with NVIDIA Quadro4 900 XGL graphics board / node

100 GB100 1.53 GHz Athlons

IA-32 Distributed parallel (128 nodes with Infiniband interconnect) and serial applications

1024 GB512 2.4 GHz Intel Pentium 4 Xeon

IA-64 Distributed parallel and serial applications

752 GB300 900 MHz Intel Itanium2

Special PurposeTotal MemoryProcessorsSystem


FPGAs Have Been Around A WhileFPGAs Have Been Around A While• FPGA computing is well established in some areas

MAPLD RECON ACM/IEEE

• Exploited by programmed logic designers in lieu of an ASIC approach General Fixed logic Reconfigurable – an option to change without physical replacement

• Utilized in many devices Vehicle systems Defense systems (radars, missiles, aircraft) Medical systems Communications hardware

…but only more recently appearing in general purpose information systems


Early Use of FPGAs in Production Systems: Early Use of FPGAs in Production Systems: TimeLogic DeCypherTimeLogic DeCypher

• FPGA for production bioinformatics Field Programmable Gate Array Deployed 2002 in SunFire 6800 enterprise systems High throughput for searching/scanning data

• Available Algorithms Tera-BLAST (N,P,X, TN, TX) Smith-Waterman HMM searches and queries Profile scan Custom target building

• The environment was closed to custom development


A Surge in FPGA Interest:A Surge in FPGA Interest:The Economic End of Moore’s Law?The Economic End of Moore’s Law?

• Clock-rate gains have slowed Power constraints Yield constraints Cost constraints

• Core limits: ‘Core-allary’: The number of

cores will double every 18 months

At what cost? For how long? Then what?

• FPGAs capabilities are still rising Gate counts Clock rates

High clock rate is a cost, not a benefit;

it drives up costs of everything else

in the machine.

-- eWeek


Businesses Challenged by Businesses Challenged by Increasing Data VolumesIncreasing Data Volumes

• Increasing amounts of data generated By research applications and instruments By transactions and transportation By sensors and cameras

• Need to store and secure more information for longer times Sarbanes-Oxley controls Protection of intellectual property HIPAA protection of health care information

• Data volumes are soaring! Example: Genbank volume doubles every 18 months Walmart preparing to add TBs regularly for RFID

• Raising needs to rapidly search, index, manage and relate data


Challenged by Power Demands for SpeedupChallenged by Power Demands for Speedup• General purpose parallel clusters eat power

For example: One Itanium II cpu requires 100 watts of power 50x speedup for a 99% efficient parallel program requires 99

processors Power demand to 9.9 kWatts! Even with more cores, power demands will increase

• The cost of power is ever increasing Energy prices rise (ala Enron) Cost of power-outages has increasingly enormous business impact Backup power must be able to maintain critical business operations

which are increasingly data intensive

• Bottom line: Power is money…out the door


Data Centers Constrained by Data Centers Constrained by Space and BudgetSpace and Budget

• Large systems need space… A hypothetical 50 node 1U system requires est. 20 square

feet of floor space for air flow One rack (dual core) delivers 100 CPU units Speedup potential of only 50 for 99% parallel app

• …for cooling systems Need for ambient air cooling increases with CPU power Multiple cores = more cooling per node

• Bottom line: Building-size refrigerators (data centers) can be very expensive to build or retrofit To build: $67M to build data center consuming 15 MW power To operate: $12M+ per year at 40% capacity


Adoption Dilemma for FPGAAdoption Dilemma for FPGA• The Application Catch-22

Few valued applications to drive market Small market for skilled developers Searching for the veritable ‘Killer App’

• Cost Benefit Equation Imbalance Cost of adoption is high

• Hardware introduction price• Application development costs• Application and system support costs• Risk of single vendor solutions


Business Challenges At HandBusiness Challenges At Hand

• Key Challenge: Absence of meaningful applications Potential is not convincing Tendency to adopt other technology solutions (clusters of cores) Cost of proven applications is high

• Potential Solutions: Educate on proven capabilities Develop the full ROI picture for FPGA solutions Deliver more applications, less expensively


Business Challenges At HandBusiness Challenges At Hand

• Key Challenge: Small pool of capable developers Limits choices for implementation Brings long-term supportability in question Cost of skills and support is high

• Potential Solutions: Enhance experience and education of developers Ease supportability of applications Wage and price controls


Complimentary Perspectives?Complimentary Perspectives?Programming vs. Hardware DesignProgramming vs. Hardware Design

EncapsulationGeneralityMaintainability

Efficient implementationImmutable definitionLow-level specification

Reliable execution

Reusability


Technical Challenges At HandTechnical Challenges At Hand

• Key Challenge: Lack of portability of invested effort FPGA algorithm implementation requires special skills Vendor solutions are generally unique

• Potential Solutions: lower amount of effort required to deliver Improve portability of elements at every level


Technical Challenges At HandTechnical Challenges At Hand

• Key Challenge: A lack of affordable, common tools Expensive tools harder to justify De facto lock-in once purchased Need to maximize return on tool investment

• Potential Solutions: Cheap tools --- in due course, when sustainable Improved portability of solutions to better maximize ROI


Technical Challenges At Hand Technical Challenges At Hand • Key Challenge: Long development times

Detailed specification may be required Range of input conditions may require multiple

independent options for execution Even Place and Route can be time-consuming

• Potential Solutions: Better development tools Mutual education – application to implementation Improved reusability of elements Adopt allowable algorithmic short-cuts


A Familiar Set of ChallengesA Familiar Set of Challenges

• Absence of meaningful applications• Small pool of capable developers • Long development times• A lack of affordable, common tools• Lack of portability of invested effort


Emerging Attractiveness of FPGA Emerging Attractiveness of FPGA Solutions Solutions

• Capability for parallel performance boost Example: Latest Virtex 4 has 6 million logic gates Demonstrated applications moving beyond illustrative examples Utilizing increasingly complex algorithms and functional units

• Easier development Many options are emerging for programming Not just VHDL anymore

• Validation More vendors buying in

• Easier introduction No remodeling to add power or cooling Single integrated solutions are emerging

• Bonus: Upside potential for increased clock speeds


encryption

simulation

annotationimage recognition

data filtering

FPGA

comparative informatics

FPGA Opportunities at HandFPGA Opportunities at Hand


OpenFPGA: Motivating ObservationsOpenFPGA: Motivating Observations

• The use of Field Programmable Gate Array technology was missing a collaborative organization focused on fostering the use of FPGA technology in high level applications.

• The results of several months of early discussions and activities involving FPGA hardware vendors, academic, government and commercial application developers, technology integrators and computer centers clarified several common needs in the FPGA application development area.

• Addressing these needs, remains inherent to successfully foster the broader and more general integration of FPGA technology in higher level and general purpose applications.


The Starting PointThe Starting Point


What We’re Trying to AvoidWhat We’re Trying to Avoid


Why OpenFPGA?Why OpenFPGA?

• Build on past successes of OpenMP and MPI to collectively foster easier application parallel solutions

• Diverse options for approaching FPGA application development and integration

• Combine efforts early key to fostering mutual evaluation and adoption of best practices

• Sooner or later, standards of interoperability needed to extend reuse and portability among FPGA applications


OpenFPGA MissionOpenFPGA Mission

“The mission of OpenFPGA.org is to promote the use of Field Programmable Gate Arrays in high level and enterprise applications by collaboratively defining, developing and sharing critical information, technologies and best practices.”

The mission is realized through pursuit of the following objective areas Innovation and Evaluation – sharing tools and best practices Standardization – collectively defining common implementations Education – developing skilled professionals Promotion – advancing successful application uses of FPGAs Communication and Collaboration – fostering communication Participation – expanding involvement


Technically, Where Does Technically, Where Does OpenFPGA Fit?OpenFPGA Fit?

• Connect Apps to FPGAs• Use common approaches

and APIs to enable mobility

• Improve supportability• Start small and focused• Build to future capabilities

Multi-vendor OS/FPGA

Environments

Apps

FPGA FPGA FPGA

Apps

open

fpga


Where Does OpenFPGA Fit?Where Does OpenFPGA Fit?• Technical possibilities

Open-source area for applications Open-source tool options Baseline standards for communication and interoperability Benchmarks

• Education Technical Public perception

• Collective progress Sharing results Debating new ideas and approaches


OpenFPGA So FarOpenFPGA So Far

• OSC took the initiative in late 2004 to form OpenFPGA following Demonstrated potential of FPGA technology in the Time Logic System Hearing common challenges expressed during several prior months of

discussion with FPGA vendors, tool providers, developers and end-users Hosting a very successful FPGA Workshop in October 2004

• Effort officially initiated by OSC in February 2005 with letters of invitation to form an exploratory OpenFPGA Steering Group 16 initial respondents (presently 19 member organizations) Cross-cutting across multiple interests, international, academic, commercial,

government

• First announced February 2005 at Manchester England Reconfigurable Computing workshop

• Co-host for RSSI held at NCSA July 2005


OpenFPGA Steering GroupOpenFPGA Steering Group

• Cray, Inc• GE Global Research• Koan Corporation• Mitrion• Nallatech• NASA• NCI-ABCC• NCSA• NIST• OSC

• Sandia National Lab• SGI• SRC• Starbridge Systems, Inc.• University of Cincinnati• University of Manchester• University of South

Carolina• University of Toledo• Zuse Institute Berlin


Current OpenFPGA ParticipationCurrent OpenFPGA Participation• More than 160 registrations representing academic,

government and commercial entities from over 11 countries

• Active in efforts ranging over Numerical simulation HPC Benchmarking and evaluation Biological modeling Clusters of FPGAs Signal processing FPGA programmability Bioinformatics algorithm acceleration Systems of linear equations High level languages Data mining Pattern matching, statistics, realtime datastream


OpenFPGA Status ReportOpenFPGA Status Report• Past

Formed steering group with regular bi-weekly call-in meetings Established mailing list to share information Created web presence at www.openfpga.org Surveyed user community for needs and interests Established forums for community discussion in objective areas

• Present First face-to-face steering group meeting on 7/15 (this week) Completion of organizational charter is top priority Identifying specific key efforts to pursue worldwide Opening participation

• Future Broadening participation BOF at SC 2005 Source to help define and evaluate standards for FPGA apps

http://www.openfpga.org/


Details on Current EffortsDetails on Current Efforts

• Finalize organizational charter to characterize the organization Develop a sustaining support model Constitute Advisory Board and Board of Directors per charter

• Prioritize developmental activities from across many potential areas Shared development tools and facilities Shared validation facilities Application specific standards Benchmark standards Application development and delivery

• Define, initiate and execute projects in common areas of collaboration, targeting community availability and impact

• Generally, demonstrate the capability of FPGA computation


Fostering Development and Use Of FPGA-based ApplicationsSign up at www.openfpga.org today.

http://www.openfpga.org/


Bridging to SolutionsBridging to Solutions

Valued Applications

Cheap Faster Solutions


• Acknowledgements Kevin Wohlever, Sharron Madero The entire OpenFPGA steering group The growing OpenFPGA community

• Thank you

• Questions

Documents

Field Programmable Gate Arrays: Power in Future HPC Systems Eric Stahlberg Ohio Supercomputer Center July 12, 2005