1

Cyberinfrastructure Begins at HomeCraig A. Stewart (stewart@iu.edu)Executive Director, Pervasive Technology InstituteAssociate Dean, Research TechnologiesAssociate Director, CREST (Center for Research in Extreme Scale Technologies)

Bradley C. WheelerVice President for Information and Technology & CIOProfessor, IU School of Business

Indiana University - pti.iu.edu

Presented at IBM Multi-customer briefing, Sheraton Hotel, Nov 15, IEEE/ACM SC11 conference, Seattle, WA

2

License Terms

• Please cite this presentation as: Stewart, C.A. and B.C. Wheeler. 2011. Cyberinfrastructure Begins at Home. (Presentation) IBM Multi-customer Briefings, IEEE/ACM SC11 Conference (Seattle, WA, 15 Nov 2011). Available from: http://hdl.handle.net/2022/13888

• Portions of this document that originated from sources outside IU are shown here and used by permission or under licenses indicated within this document.

• Items indicated with a © are under copyright and used here with permission. Such items may not be reused without permission from the holder of copyright except where license terms noted on a slide permit reuse.

• Except where otherwise noted, the contents of this presentation are copyright 2011 by the Trustees of Indiana University. This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.

3

Introduction and Motivation• “If you don't know where you are going, any road will get you there” – Lewis

Carroll

• If you want to go somewhere, set a strategy and pursue it with diligence

• IU wants to go somewhere:

– To be one of the great public universities of the 21st Century (Michael A. McRobbie, 18th President of IU)

– To be a leader, “in absolute terms for uses and applications of IT” (Myles Brand, 16th President of IU)

• What is cyberinfrastructure, anyway?

– First used in security briefing by Richard Clark in 1998.

– Cyberinfrastructure consists of systems, advanced instruments and data repositories, visualization environments, and people, all linked together by software and high performance networks to improve research productivity and enable breakthroughs not otherwise possible.

4

Outline

• Some background about Indiana University (fairly extensive)

• Where we are now

• How we got here

• Where we think we are going

• Generalizations and portable lessons

5

IU – Campuses and Medical School Centers

IU Campuses IU School of Medicine campuses and clinics

6

IU – Two Core Research/Education Campuses, Six Regional Campuses

Campus Academic appointees

Staff (non-academic)

Undergraduate Students

Graduate & Professional Students

IUB 2,973 5,442 32,367 10,097IUPUI 3,847 4,793 22,245 8,321IU Northwest 393 245 5,307 662IU South Bend 559 315 7,790 800IU East 225 152 3,285 80IP Fort Wayne 838  736  12,500  800 IU Kokomo 167 142 2,978 131IU Southeast 485 250 6,384 794Totals  8,649  11,339  80,356  20,885

IU community: 121,229 people total

7

IU Budget Category 2010/2011 BudgetUnrestricted $2,002,537,391Restricted $615,974,635Auxiliary $407,445,800Total $3,025,957,826

IU Health Patient Metrics – 2010/11

Admissions 115,250Outpatient visits 1,882,795Staffed Beds 2,889

Key IU Metrics

8

IU’s Goal – Be One of the Great Public Universities of the 21st Century

As a great public university we have two fundamental missions: education and research. Indiana University seeks to provide the best possible education to all of our students, both undergraduate and graduate. It is an education in both breadth and depth, grounded in both the practical and the transcendent, and providing a command of the analytical and the expressive. We also seek to conduct path-breaking research and scholarship of the highest international standards from the most theoretical to the most applied. …

We are also a public university supported by and with a responsibility to the citizens of Indiana. They expect us to provide a great education to their sons and daughters; they expect us to do the best research and scholarship; and they expect us to be engaged in the life of the State. Therefore, engagement is a third mission, and it grows out of excellence in education and research.

-IU President Michael A. McRobbie

McRobbie, M.A. 2008. Inaugural address of Michael A. McRobbie: Endurance, Excellence, and the Energy of Change at Indiana University. http://www.indiana.edu/~pres/speeches/101807.shtml

And … the University Campuses and Culture Are Great9

http://bloomsofbloomington.org/

http://www.iupui.edu/galleries/272/ndAnnualIUPUIRegattaPhoto by John R. Gentry © Trustees of Indiana University

http://www.sequenza21.com/2011/04/the-premiere-of-bernard-rands%E2%80%99s-vincent-operatic-meditations-on-the-life-of-an-artist/

10

11

12Begin with the End in Mind: End Goals for Service –

IU Researchers, Scholars, Artists, and Students

• For IU researchers, scholars, students & artists:

– To use information technology within a policy and security framework that is a model for academia and the US;

– pursue academic and creative activities without limitations created by resources for data management, computation, or visualization, accessible with state-of-the-art ease of use;

– examine and present research data, simulation results, or artistic creations intuitively to enhance scholarly and artistic efforts;

– and have resources available 24 x 365, immediately when desired.

13

Begin with the End in Mind: End Goals for Service – Indiana Residents

• For Indiana residents:

– To have the benefits of information technology services and information available from IU;

– the opportunity to obtain a first-class education at all levels; the social and economic benefits of having IU produce talented graduates, well educated and ready to pursue interesting and valuable careers;

– a vibrant economy providing satisfying, high-quality jobs with good pay that provide new career options and entice IU graduates to stay; a high standard of living and quality of life;

– and an engaging and enjoyable cultural community.

14

Begin with the End in Mind: End Goals for Service – Others in the US and the World

• For others in the United States and the world: – To have the benefits of information technology services and IT-related

information from IU; – improved quality of life as a result of economic advancement, improved

health, and artistic and literary creations brought about by the leadership of IU and the state of Indiana.

15Goals for 2019• IU researchers, scholars, students, and artists will

– Use information technology within a policy and security framework that serves as a model for academia and the US in general.

– Be able to pursue their academic and creative activities with no limitations created by access to data, and few limitations caused by access to computational power.

– Be able to examine and present research results or artistic creations in ways that are intuitive and enhance effectiveness through

• 2- and 3-D display and interface resources generally available in offices, labs, and meeting rooms.

• State-of-the-art large-scale facilities located conveniently throughout IU.– Have access to resources that

• Grow in capability and capacity predictably, steadily, and in ways that keep IU researchers at the leading edge of discovery.

• Are available resiliently by design (24 x 365 at never less than 75% aggregate capacity).

• Are available immediately when immediacy is essential.• Are accessible through interfaces that are intuitively usable by the large

majority of IU researchers.

16

Indiana residents will

• Have the benefits of information technology services and IT-related information made available to state residents by IU.

• Benefit from new, high-quality jobs created by IU’s advanced IT environment (at rates exceeding the present rate of job creation and contributing strongly to the Indiana economy). Such jobs will be created in three ways:

– Bring federal money into the state to create new jobs.

– Attract existing companies to locate major business operations in Indiana.

– Create new companies through commercialization of innovations from IU.

• Have available education and training so anyone growing up in Indiana can strive for and obtain one of these high-quality jobs. The School of Informatics and Computing and other IU schools matriculate well-educated graduates, many of whom stay in Indiana.

• Have an improved quality of life stemming from these achievements:

• Indiana will rise from 23rd in 2008 to at least 18th in overall rank in the AeA’s annual Cyberstates report.

• UITS and PTI will aid IU biomedical research and health services communities to improve state rankings in major health indicators like obesity and tobacco use.

• US and world residents will benefit through access to information technology services and IT-related information made available by IU; and have improved quality of life, enabled at least in part by the outcomes of IU discoveries and innovations.

Goals for 2019, con’t

17Research-oriented Cyberinfrastructure, Informatics,

and Computer Science Units• UITS

– Research Technologies Division– Networks / GlobalNOC– Portions of Enterprise Infrastructure– Portions of Support– Portions of Learning Technologies– Portions of Enterprise Software

• Academic units– School of Informatics and Computing– Maurer School of Law– College of Arts and Sciences

• Pervasive Technology Institute – collaboration involving OVPIT, SOIC, Maurer School of Law, College of Arts and Sciences, UITS

• Key strategies:– Competition is never within IU, and usually outside the state of Indiana– Research cyberinfrastructure is a non-zero-sum game from the

standpoint of the university

18

• IU Bloomington Data Center designed to withstand category 5 tornado.

• Informatics & Communications Technology Complex houses IUPUI offices and Data Center

• Innovation Center, dedicated in November 2009, provides a central location for collaboration between discipline scientists, computer scientists, and staff through the Pervasive Technology Institute

• Cyberinfrastructure Building houses OVPIT, UITS staff. To be Gold (or higher) LEED certified

Machine room total ft2

Avail. ft2 Power total

Net power avail.

Cooling capacity total (tons)

Cooling capacity avail. (tons)

ICTC 8,300 1,400 600 kW 208 kW 290 150

IUB Data Center

30,000 15,000 1.46 MW 592 kW 2200 550

CI Facilities

21

Computational Systems

Name Architecture TFLOPS Total RAM (TB)

Local disk (TB)

Big Red IBM PowerPC 970MP (JS21 blades)

40.96 8 73

Quarry IBM e1350 Intel Xeon (HS21 blades)

8.96 1.2 5

Mason HP DL580 G7 Intel Xeon servers 3 8 16

RDC IBM Power5 (p575) database servers, Intel Xeon E5310

N/A 0.14 50

FutureGrid IBM e1350 12 6 128

FutureGrid CrayXT5m 6 1.3 5.5

Totals    79.29 15.84 211.7

22

Storage Systems

Name Architecture Disk (PB) Tape (PB)

Research File System OpenAFS 0.06 NA

Data Capacitor Lustre 0.94 NA

Scholarly Data Archive HPSS 0.25 5.7

Totals    1.23 5.7

23

An Array of Viz Systems

Immersive Theaters Ultra-Resolution Stereoscopic

IUB

I

UP

UI

24

A Few of IU’s Major Funded Activities

• XSEDE (Stewart, local PI)

• FutureGrid (Fox, PI)

• SEAD (Plale, local PI)

• Hathi Trust Research Center (Plale, PI)

• PXGL, ParallaleX, BOOST Graph library (Sterling, Lumsdaine local leads)

• Indiana CTSI (Barnett, IT lead)

• Operation Ice Bridge (Knepper, local lead)

• OGCE (Pierce, PI)

25The Local Community is Happy – Spring 2010 User Satisfaction Survey

How helpful has the information technology environment at IUB been in your research activities?

Overall, how satisfied are you with the UITS research technology services available at IUB?

Average Satisfaction Usage

Central research and high performance computers (Big Red, Quarry, and RDC clusters) [F, Staff, G] 4.16 ± 0.08 92.9 ± 2.2% 14.6%

Center for Statistical and Mathematical Computing (StatMath Center, statmath@indiana.edu, 855-4724) [All] 4.08 ± 0.06 94.0 ± 1.7% 20.1%

Massive Data Storage Service (MDSS/HPSS) [F, Staff, G] 4.16 ± 0.07 93.5 ± 2.1% 16.2%

Advanced Visualization Laboratory (AVL); www.avl.iu.edu [F, Staff, G] 4.32 ± 0.07 97.8 ± 1.3% 7.8%

Support for software applications using IU’s high performing computing resources [F, Staff, G] 4.11 ±0.08 93.3 ± 2.1% 15.2%

Support for life sciences research (Bioinformatics, The Center for Computational Cytomics [F, Staff, G] 3.91 ± 0.08 91.5 ± 2.3% 8.3%

Average Satisfaction Usage

4.17 ± 0.06 95.5 ± 1.5% 50.0%

Average Satisfaction Usage

4.19 ± 0.09 95.9 ± 2.2% 83.9%

26

We’ve Come a Long Way

• In 1997, when IU had its first display at SCxy, the most common question we heard was “Why does IU have a booth at the supercomputing conference?”

• So how did we do it?• 1998 Indiana University Strategic Plan

(Architecture for the 21st century) adopted• 10 Recommendations, 68 Actions• Mostly focused on technology stacks and

the spectra from “competent to excellent” and “no longer needed to phased out”

27Year Notable Events, Grants, and Awards1955 Marshall Wrubel named first permanent director of Research Computing Center1964 Frank Prosser, Stan Hagstrom, Steve Young publish FASTRAN compiler

1970s IU develops significant CDC environment, major location for BMDP ports from IB => CDC

1981 Bloomington Academic Computing Services formed, first full time administrator (Daniel W. DeHayes, Business Professor)

1990s IU major VAX facility, focus on computing in humanities, some significant scientific computing

1997 Michael A. McRobbie comes to IU as first full VP for Information Technology and CIO1997 vBNS Grant – Connected IU to NSF-funded high speed Internet ($394,000)1997 First IU display at the international SC conference on Supercomputing1998 IU IT Strategic Plan adopted 1998 Digital Library IBM SUR (Shared University Research) grant – hardware to support the

Variations digital music library project1998 GRNOC: Internet2 network engineering and operations support ($3,700,000 contact)1999 Lilly Endowment, Inc. funds Pervasive Technology Labs ($29M) (Fox recruited)1999 Indiana General Assembly funds I-Light Network ($5.300,000)1999 NSF funds network connectivity to Asia (TransPAC) ($10,100,100)

28Year Notable Events, Grants, and Awards2000 NSF funds Variations2 digital music library project ($3,100,000M)2000 Research @ Indiana Display at SC2000 – first time at this conference that all of the

major research universities of any state coordinate a display promoting their home state as a whole

2000 Lilly Endowment, Inc. funds Indiana Genomics Initiative ($105,000,000 total; $6,700,000 for advanced Information Technology)

2001 John-E-Box invented2001 IU announces first 1 TFLOPS supercomputer owned by a US university; it appears as

the 50th fastest supercomputer on the on Top500 list in the world in November2001 I-Light activated on December 112001 NSF funds AVIDD, first university-owned teraflops supercomputer ($1,800,000)2001 GRNOC: Indiana Gigapop supporting Higher Education access to national R&E

($500K)2002 IU establishes first distributed, disaster-resilient tape archive running under the highly

secure HPSS (High Performance Storage Software). Data written to data in tape silos simultaneously in Indianapolis and Bloomington to ensure reliability of data preservation

2002 IU joins in life sciences initiative as part of Central Indiana’s life sciences corridor

29Year Notable Events, Grants, and Awards2003 AVIDD achieves more than 1 Trillion calculations per second on standard benchmark

application – the First distributed supercomputer cluster to surpass the 1 TFLOPS barrier. AVIDD appears in 50th place on Top500 list in June.

2003 NSF funds IP-grid project providing $1,500,000 for IU to become part of TeraGrid, and $1,400,000 to fund Purdue’s participation

2004 Lilly Endowment, Inc. funds Indiana METACyt Initiative ($53M total; $6.25M for life sciences IT)

2004 Informatics and Communication Technology Complex Building dedicated on IUPUI campus – new home for School of Informatics, UITS, and Pervasive Technology Labs

2005 NSF funds IU’s early operations participation in TeraGrid ($440,000)2005 NSF funds IU as a TeraGrid Resource Partner ($4,100,000)2005 NSF funds network connectivity to Asia (TransPAC2) ($5.1M)2005 GRNOC: National Lambda Rail engineering and operations support ($7.5M)2005 NSF funds Data Capacitor – very fast data storage system ($1,720,000)2006 Big Red: fastest academic supercomputer in western hemisphere, 23rd fastest

supercomputer in the world on the June Top500 list.2006 IEDC/IBM grant to double Big Red. Big Red listed as ###___ fastest supercomputer in

the world on Top500 list.2006 I-Light: First customer connection to backbone ($15.5M)

30Year Notable Events, Grants, and Awards2007 NSF funds PolarGrid ($1.9M) (Fox PI)2007 NSF funds network connectivity to Pakistan ($950K)2007 Bandwidth Challenge award – IU moves data across international and national networks

faster than any other supercomputer center competing in this contest at the 2007 Supercomputing Conference

2008 Lilly Endowment, Inc. funds Pervasive Technology Institute ($15M)2008 I-Light: Network backbone complete ($4.9M)2009 IU research systems approved by IU Counsel – meets standards for alignment with

HIPAA regulations. IU researchers now able to use IU research systems to analyze electronic protected health information, thus speeding research

2009 IU Professor Alex Vespignani uses Big Red and I-Light to predict spread of H1N1 virus and transmit that data to federal agencies in Washington

2009 NSF funds GRNOC: Global Environment for Network Innovations ($330K)2009 NSF funds FutureGrid ($10.1M) – the fourth of the so-called Track I and II systems

announced officially by the NSF. IBM, Cray major partners2009- 2010

Stewart chairs NSF ACCI Task force on Campus Bridging

2011 IU $4.9M Subcontract as part of XSEDE2011 NSF funding $1.5M for National Center for Genome Analysis Support2011 CREST opens – Center for Research in Extreme Scale Technologies2011 IU announces partnership with Penguin Computing, Inc. for cluster as a service

31

Drivers of Consolidation in the Past (and Ongoing)• Competence and trust

– Summaries of all survey data and every comment (minus obscenities and identifying references) from 1986 on is available on the web

• Facilities and gear– Especially systems obtained through collaborative grants

• Mission orientation– Constantly asking: what is the mission of IU – are we following it– We ask the faculty and students to trust our competence…. We trust

their scholarly and artistic leadership– Serve the home crowd first, the national community when it reinforces

service to the home community. And serve all of the home crowd– Vendor partnerships. The best partnerships have been based on

complementary strengths and shared goals, and those that persisted even when there was no money on the table

32European Finances and Isaac Newton

Campus Scholarly Infrastructure

Domain SpecificDiscovery & Innovation,

Teaching & Learning

Necessary Infrastructure Leveraged

Discipline Research Stacks…

Innovation

Metadata

Computation, Storage

Models

Networks

Genomics

Visualization

Curation

Anthropology

Searching & Retrieving

Networks

Metadata

Curation

Innovation,Publication

Storage

Visualization

Arts

Networks

Curation

Storage

Metadata

Retrieval &Analysis

Innovation

Primary Storage

Physics

Distributed Storage

Networks

Metadata

Models

Computation

Visualization

Innovation

Shared Cyberinfrastructure Line Here?

Shared Cyberinfrastructure Line Here?

© Brad Wheeler, Indiana University, Creative Commons Attribution License 2.5

34

Efficiencies and Risk Profile

• Larger buildings and larger air conditioners more cost-effective per unit cooling – savings of more than 50% possible

• Hiding costs by using Graduate Assistants as sysadmins may increase costs:

– Cost of decreased progress on degree by GA

– How good are most GAs as sysadmins anyway?

• Cluster in a closet - Cost of down time and underutilization (in exchange for individual-driven queue policy

• Network monitoring, electronic security, and physical security

• Hacker, T.J., and B.C. Wheeler. 2007. Making Research Cyberinfrastructure a Strategic Choice. EDUCAUSE Quarterly 30 (1): 21-29

35

Money Matters

• Money is everything (or money can be converted into most anything). Or: Money is distilled personality

– We know how our money is spent: Activity Based Costing metrics. We know what our services are, and what the aggregate and units costs of each service are

• Our approach has been to build a philosophy of abundance in systems, people, pay for the people, travel, and opportunities for innovation [No or minimal internal nuisance fees]

• Hardware can be a good substitute for money early on if you can get it and are careful to pay for people to support the hardware and its users

• Much of IU’s success in many dimensions comes from combining hard-to-combine streams of money

36

Some of the 7 Laws of Money (Michael Phillips, Shambala Sun)

• 1. Do it! Money will come when you are doing the right thing.

• 3. Money is a dream

• 4. Money is a nightmare

• 6. You can never really receive money as a gift.

• Why do we (OVPIT/PTI/RT) pursue grant awards?

– In the sincere belief that we are better suited to do some particular task than anyone else in the world and that the world will benefit from us doing those tasks

– In the belief that pursuit of a grant award will provide local as well as national benefits

37

Positive Feedback Loop

Graphic by Bradley C. Wheeler. Used under Creative Commons 3.0 unported attribution license

38Staff Funding Sources

_x0002_96

_x0002_97

_x0002_98

_x0002_99

_x0002_00

_x0002_01

_x0002_02

_x0002_03

_x0002_04

_x0002_05

_x0002_06

_x0002_07

_x0002_08

_x0002_09

_x0002_10

-

20

40

60

80

100

120 PTI Soft

PTI Base

RT Soft

RT Base

Fiscal Year

FT

E

39

Edge EdgeTrustTrust

Leverage

Evolution of IT Services

? ?

•15 Recommendations

•72 Action Items

2008 – IU IT Strategic Plan

• 15 Recommendations

• 72 Action Items

• Role and function based, not technology based

40

Empowering People – Recommendations Relevant to Research

• Recommendation 1: Indiana University's national and international leadership should be sustained through continued maintenance and advancement of an IT infrastructure that is supported by sound fiscal planning.

• Recommendation 9: Indiana University should provision appropriate "data utilities" for administrative data/information, research data, teaching and learning resources, and multimedia scholarly life. These utilities should provide convenient, timely, and secure access to university data/information by the IU community and authorized collaborators beyond IU.

• Recommendation 15: While Indiana University should advance IT-enabled research across all disciplines, it should also focus on a few highly promising opportunities for which it has a skills, knowledge, and reputational advantage to push the frontiers of IT-enabled research and scholarship.

41

Factors Driving Consolidation within the University in the Future

• Economies of scale in management

– Air conditioning

– Electricity supply

– Staff

• Security

• VM hosting – intelligent infrastructure

• Economies of vendor partnerships and large purchases

• Redundant backups (2 locations) and our own network

42

Intelligent Infrastructure• Virtual systems supply the infrastructure and network capacity necessary to

host your applications, while optional disk storage on UITS enterprise-class SANs (storage area networks) ensures your files are extremely secure and always available.

• Backup solutions provide cross-site backups and cross-campus failover options, which isolate you from potential disasters by securing your backup files within hardened data centers.

• This is a charge-back solution - $425 / system / year; $1 /GB / year

• Adoption and growth have been phenomenal – service introduced in July 2007, set up 1000th VM in 2009, today have a total of 1981.

• Tremendous energy efficiencies - 26 VMs per physical core

• Departmental staff are able to focus more on things that are unique to their departments – and with implementation of CMS(es), IU’s web presence is improving

43

IUanyWARE – Core Attributes

• Commodity delivery of IU applications (> 200)• Target Community: IU Staff, Students and Faculty• Generic easy to consume features• OS agnostic

• Android• Chrome• iOS• Windows• Linux

• Access to IU software anywhere anytime and on any device for 100,000 students and over 17,000 full-time staff.

• Allows virtual consolidation of service delivery without physical consolidation of students in student computing clusters!

44

PTI - History and Goals • Started by $30M grant from the Lilly Endowment to create the Pervasive

Technology Labs and fuel growth of the IU School of Informatics and Computing. 2nd round funding of $15M to create Pervasive Technology Institute in 2008

• Overall: – Invent, develop, deploy, deliver – Serve IU’s flagship ‘brand’ re this activity in advanced IT

• PTI Research Centers Goal:– Research Excellence

• PTI Service & Cyberinfrastructure Centers Goals:– Support Development, deployment, support of research IT– Support research-related activities by IU community generally, PTI

Research Centers particularly, and national research community when in IU’s interests

– Aid state of Indiana• Baseline level of workforce & economic development activities

45

PTI Structure

• PTI is a collaboration among School of Informatics and Computing; Maurer School of Law; Office of the Vice President for Information Technology; University Information Technology Services

• Consists of two types of centers:

– Research Centers: D2I; CACR; DSC; CREST. Needs and communication coordinated by Managing Director.

– Service/Infrastructure Centers: Research Technologies Division of UITS. Needs and communications coordinated by Executive Director

• Managing Director and Executive Director report to VP for IT and are jointly responsible for success of PTI

• Executive Committee: Relevant Deans, VP for Engagement, VP for IT

• Reports administratively to OVPIT

46

• 437 full time job-years of employment created within IU by grants brought into Indiana since 1999

• Based on economic multipliers, 2,705 full time job-years of employment in the state of Indiana since the start of PTL in 1999

Economic Development

2002 2004 2008 2010

Indiana 30 29 33 28

Illinois 19 21 21 20

Kentucky 46 48 47 47

Michigan 24 25 26 26

Ohio 27 24 36 29

Milken Institute’s State Technology and Science Index for Indiana and bordering states

47

Cyberinfrastructure Begins at Home, But it Doesn’t End There

48

Commercial cloud (Iaas and Paas)

Volunteer computing

Workstations at Carnegie research universities

Campus HPC/ Tier 3 systems

Track 2 and other major facilities

NSF Track 1

0 2,000 4,000 6,000 8,000 10,00012,000TFLOPS

Based on: Welch, V.; Sheppard, R.; Lingwall, M.J.; Stewart, C. A. 2011. Current structure and past history of US cyberinfrastructure (data set and figures). http://hdl.handle.net/2022/13136

Some CI Resources Available to Science and Engineering Researchers in the US (March 2011)

49Adequacy of Research CI

Never (10.6%)

Some of the time (20.2%)

Most of the time (40.2%)

All of the time (29%)

Stewart, C.A., D.S. Katz, D.L. Hart, D. Lantrip, D.S. McCaulay and R.L. Moore. Technical Report: Survey of cyberinfrastructure needs and interests of NSF-funded principal investigators. 2011. http://hdl.handle.net/2022/9917

Responses to question asking if researchers had sufficient access to cyberinfrastructure resources – survey sent to 5,000 researchers selected randomly from 34,623 researchers funded by NSF as Principal Investigators 2005-2009; Results based on 1,028 responses.

50

Local and National: Use Facilitated

• XSEDE – 11 Projects, > 4M TeraGrid normalized units / year

• Open Science Grid

• ~$1M/year in resources for IU researchers

51

Photo by http://www.flickr.com/photos/mnsc/http://www.flickr.com/photos/mnsc/2768391365/sizes/z/in/photostream/

http://creativecommons.org/licenses/by/2.0/

Clouds Look Serene Enough

52

But is Ignorance Bliss?

• In the cloud, do you know:

– Where your data are?

– What laws prevail over the physical location of your data?

– What license you really agreed to?

– What is the security (electronic / physical) around your data?

– And how exactly do you get to that cloud, or get things out of it?

– How secure your provider is financially? (The fact that something seems unimaginable, like cloud provider such-and-such cloud provider going out of business abruptly, does not mean it is impossible!)

• NB: Thomas Friedman wrongly insulted the state of Indiana

53

There Are Good Definitions of Cloud Computing - NIST

• Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

• This cloud model promotes availability and is composed of five essential characteristics (On-demand self-service, Broad network access, Resource pooling, Rapid elasticity, Measured Service); three service models (Cloud Software as a Service (SaaS), Cloud Platform as a Service (PaaS), Cloud Infrastructure as a Service (IaaS)); and, four deployment models (Private cloud, Community cloud, Public cloud, Hybrid cloud).

• Key enabling technologies include: (1) fast wide-area networks, (2) powerful, inexpensive server computers, and (3) high-performance virtualization for commodity hardware.

• http://www.nist.gov/itl/cloud/index.cfm

54

Above Campus Services• "We are seeing the early emergence of a meta-university — a transcendent,

accessible, empowering, dynamic, communally constructed framework of open materials and platforms on which much of higher education worldwide can be constructed or enhanced.” Charles Vest, president emeritus of MIT, 2006

• “Shaping the Promise of Cloud Computing for Higher Education” – Wheeler and Waggener

• Goal: achieve economy of scale and retain reasonable measure of control• Examples:

– Internet2 and HP– IU, UM, UVa, Penguin Computing Inc.

• See: Brad Wheeler and Shelton Waggener. 2009. Above-Campus Services: Shaping the Promise of Cloud Computing for Higher Education. EDUCAUSE Review, vol. 44, no. 6 (November/December 2009): 52-67. http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume44/AboveCampusServicesShapingtheP/185222

In early 2009 National Science Foundation’s (NSF) Advisory Committee for Cyberinfrastructure (ACCI) charged six different task forces to make strategic recommendations to the NSF in strategic areas of cyberinfrastructure:

• Data• Grand Challenges and Virtual

Organizations• High Performance Computing• Software and Tools• Work Force Development• Campus Bridging

Campus Bridging55

56

Highlights of Campus Bridging Taskforce Report Conclusions

• Stop the madness – just adopt InCommon / XSAML certificates

• NSF must lead the community in establishing a blueprint for a National CI

• Every institution of higher education should have a strategic plan for IT

• The NSF should fund activities that support the evolution and maturation of cyberinfrastructure through careful analyses of needs (in advance of creating new CI facilities) and outcomes (during and after the use of CI facilities)

• pti.iu.edu/campusbridging

57

NCGAS and Campus Bridging

• The National Center for Genome Analysis Support

• A Cyberinfrastructure Service Center affiliated with the Pervasive Technology Institute at Indiana University (http://pti.iu.edu)

• Dedicated to supporting life science researchers who need computational support for genomics analysis

• Initially funded by the National Science Foundation Advances in Biological Informatics (ABI) program, grant # 1062432

• Provides access to genomics analysis software on supercomputers customized for genomics studies including IU Mason system, POD IU, Stampede, Gordon, and other XSEDE systems

• Particularly focused on supporting genome assembly codes such as:

– de Bruijn graph methods: SOAPdeNovo, Velvet, ABySS,

– consensus methods: Celera, Newbler, Arachne 2

• For more information, see http://ncgas.org

58Some Things We Really Got Right from the Start

• Key assets for any university: faculty

• Key assets for the IT organization: reputation for good character, people, then everything else

• It takes a lot of Ph.D.s, and a mix of domain scientists and computer scientists

• Put health and family first, no matter what, at least in times of crisis

• Mix of funding sources

• Publish, even if we wouldn’t perish otherwise

• Participate in community, partner effectively

• Engage faculty as partners and leaders

• Timing, some days (SC, challenges, Top500)

• Truth is a wonderful thing; no cynicism zone

• Lots of feedback – difference between input rights and decision rights (and sometimes input rights don’t exist)

• Understanding how it is we do economic development

59

And Some Things We Didn’t Get Right to Begin With

• Leading where faculty aren’t

• FIFO

• Believing we had to put up with staff who didn’t want to be part of our team because they said they were unique and we had to

• Reading Sun Tzu without Lao Tzu, and vice versa

• Believing too much that if we paid attention to 4-space, press-space would take care of itself

• Listening to funding agencies too much

• Work overload, insufficient attention to life balance and creativity

• If I (Stewart) had one thing to do over … we would have done fewer projects each at larger scale early on

60

And Things We Think We Have Learned and Now Internalized

• Play to win, or not at all

• The community is too good for any one group to win very often, unless you’re very very good and have lots of money to pitch in

• Being good is often not enough

• One time money is too easy to get without base

• Heroic effort is not a sustainable strategy

• Partnering and openness

• Don’t hone in on bad ideas too quickly (but be careful of the side effects in terms of project thrash)

61

Key General Lessons

62

Leadership Matters

Myles BrandIndiana University

President 1994-2002

Michael A. McRobbieIndiana University

VP for IT & CIO 1997-2007President 2007-

… and teamwork matters too!

63

Assertions about CI at Home and Beyond1) Well-tuned campus CI enables greater research outcomes

2) Well-tuned campus CI is very expensive …though less expensive than other models

3) Well-tuned CI is possible only in the presence of clear strategy at the institutional level

4) Campus CI is (mostly) a coordination problem in the guise of a money problem

5) Above campus CI is not magic, and does not take away responsibility for delivering service to local CI users… but properly employed can be a useful strategic tool

6) Leadership in service of mission is the essential means of creating sustainable campus CI

7) Effective partnership is essential – colleagues, community, institutions, vendors

64

If I Were Starting from Scratch or Nearly So…• Find some faculty advocates and cultivate relationships with them• If you are at a place with good leadership => then educate leadership, build

consensus for some small pilot projects among faculty• If you are at a place without good top level leadership (and that happens) =>

then work directly with faculty to build consensus for some pilot projects• Build trust• Take some risks. If the organization you work for won’t support you when

you take reasoned risks and your only success is a learning experience, you deserve a better organization as your employer.

• If you don’t have anyone on your staff with a Ph.D., find someone with a Ph.D. (not in CS) and a good customer service attitude and hire them.

• Assign one person the full time job of supporting people in use of XSEDE and Open Science Grid – particularly the process of applying to XSEDE

• Pay close attention to public relations, but lead with the scholarly results and why they matter, not where you are on the Top500 list

65

Get Some Facts and Have Fact-based Discussions

• Pay someone else to do a real, human-subjects approved survey of your clients and repeat it every year

• Initiate a service definition and Activity Based Costing exercise• Publish, including effective self-publishing:

– NSF-format scholarly nuggets– Technical reports– Use the nearest real digital repository and a print on demand service

(we use CreateSpace)• Communicate concisely, well, and in English. Hemingway is easier to

process than Faulkner• Make the economic and workforce case to your community and state • Remember highlights of your history and repeat them often• Remember lowlights of your history, study and understand then, remember

the lessons for you and your organization, but don’t proactively remind the people you serve about them

• Study history of leadership, contention, and conflict

66Work This Cycle – Even If You Just Start With a Small

Hardware Grant-in-Kind

Graphic by Bradley C. Wheeler. Used under Creative Commons 3.0 unported attribution license

NB: Of the things we have done, the cyberinfrastructure service over the years hasbeen the easier. The development of PTL and PTI has been by far the harder. Some Institutions that have not done well have started with the research center first, beforeEstablishing a base of trust and good will with general service to the community.

67Participate in Community• The definitions of friends and colleagues are different. An organization is

effective when colleagues collaborate effectively.• Remember who your competition is (and isn’t)• Collaborate well, and maintain strong partnerships with other academic

organizations and vendor(s). • Participate in community

– Volunteer for conferences, serve on review panels– Join appropriate groups (CASC perhaps)– Submit benchmarks to SPEC, HPCC, Graph500, Green500– Join InCommon or at least set up a mechanism to deliver InCommon-

approved XSAML credentials when needed– Get someone to become an XSEDE Campus Champion (preferably

outside IT shop)– Make your materials reusable under a sensible license (Creative

Commons for docs / some reasonable public license for data)– Remember credit is a nonconserved property

68Alternate Financial Models• IU operates on Responsibility Center Management with a non-optional

financial allocation for certain central services … including IT• IU has also benefitted greatly from support of the Lilly Endowment• The relationship models IU has developed are likely fairly portable• Financial models at different universities vary• If you want to look at different financial and organizational models, I

suggest:– Purdue – the best current example of a community funding model (Penn

State was the pioneer in this area)– Texas Advanced Computing Center – they do things big in Texas, but

the model of excellent science and strong relations to alums and donors might be more transportable than one would imagine at first blush

– The work that Amy Apon did at University of Arkansas bears note as well – basically building on expertise and a Major Research Infrastructure award from the NSF to bootstrap a small and very good center into existence

69

"The struggle itself...is enough to fill a [person’s] heart. One must imagine Sisyphus happy.”

–Albert Camus

And Remember That We Are Living in the Myth of Sisyphus

Sisyphys (1548-1549) by Titian, Prado Museum, Madrid, Spainhttp://en.wikipedia.org/wiki/File:Punishment_sisyph.jpgThis work is in the public domain in the United States, and those countries with a copyright term of life of the author plus 100 years or fewer.

70

Absolutely Shameless Plugs

• XSEDE12: Bridging from the eXtreme to the campus and beyond• July 16-20, 2012  |  Chicago• The XSEDE12 Conference will be held at the beautiful Intercontinental

Chicago (Magnificent Mile) at 505 N. Michigan Ave. The hotel is in the heart of Chicago's most interesting tourist destinations and best shopping.

• Watch for Calls for Participation – coming early January

• And please visit the XSEDE and IU displays in the SC11 Exhibition Hallway!

71Thanks• IBM for a long, and VERY productive, collaboration

• All of the IU Research Technologies and Pervasive Technology Institute staff who have contributed to the development of IU’s advanced cyberinfrastructure and its support

• All of our faculty collaborators, especially the PTI-affiliated faculty

• Thanks to the colleagues who have contributed directly to this presentation: Malinda Lingwall, Vince Cannon, Greg Moore, Rob Lowden, Duane Schau, Eric Wernert, Mike Boyles, Bill Sherman

• Those involved in campus bridging activities: Guy Almes, Von Welch, Patrick Dreher, Jim Pepin, Dave Jent, Stan Ahalt, Bill Barnett, Therese Miller, Malinda Lingwall, Maria Morris, Gabrielle Allen, Jennifer Schopf, Ed Seidel

• NSF for funding support (Awards 040777, 1059812, 0948142, 1002526, 0829462, 1062432, OCI-1053575 – which supports the Extreme Science and Engineering Discovery Environment)

• Lilly Endowment, Inc. and the Indiana University Pervasive Technology Institute

• Any opinions presented here are those of the presenter and do not necessarily represent the opinions of the National Science Foundation or any other funding agencies