8 June 2012

What is campus bridging and why should XSEDE Campus Champions care?Craig Stewart (stewart@iu.edu)

Executive Director, Pervasive Technology Institute, Indiana University and XSEDE Campus Bridging Manager

Please cite as:

Stewart, C.A. What is campus bridging and why should XSEDE Campus Champions care? 2012. Presentation. Presented at: Campus Champions virtual meeting (Michigan City, IN, 8 Jun 2012). http://hdl.handle.net/2022/14709

All slides (except where explicitly noted) are copyright 2011 by the Trustees of Indiana University, and this content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/)

The beginnings of all this….

• In early 2009 National Science Foundation’s (NSF) Advisory Committee for Cyberinfrastructure (ACCI) charged six different task forces to make strategic recommendations to the NSF in strategic areas of cyberinfrastructure: Campus Bridging; Data; Grand Challenges and Virtual Organizations; High Performance Computing; Software and Tools; and Work Force Development.

• Cyberinfrastructure consists of computational systems, data and information management, advanced instruments, visualization environments, and people, all linked together by software and advanced networks to improve scholarly productivity and enable knowledge breakthroughs and discoveries not otherwise possible.

• The goal of campus bridging is to enable the seamlessly integrated use among a scientist or engineer’s personal cyberinfrastructure; cyberinfrastructure on the scientist’s campus; cyberinfrastructure at other campuses; and cyberinfrastructure at the regional, national, and international levels; as if they were proximate to the scientist. When working within the context of a Virtual Organization (VO), the goal of campus bridging is to make the ‘virtual’ aspect of the organization irrelevant (or helpful) to the work of the VO.

3

http://pti.iu.edu/campusbridging/

4

Branscomb Pyramid

5

NSF Advisory Committee for Cyberinfrastructure Task Force on Campus Bridging. Final Report. March 2011. http://www.nsf.gov/od/oci/taskforces/TaskForceReport_CampusBridging.pdf

6

Commercial cloud (Iaas and Paas)

Volunteer computing

Workstations at Carnegie research

universities

Campus HPC/ Tier 3 systems

Track 2 and other major facilities

NSF Track 1

0 2000 4000 6000 8000 10000 12000

Not a Pyramid - Estimated Computing Capacity (TFLOPS)

From Welch, V., Sheppard, R., Lingwall, M.J., Stewart, C.A. 2011. Current structure and past history of US cyberinfrastructure (data set and figures). http://hdl.handle.net/2022/13136

Key initial findings of NSF ACCI Task Force on Campus Bridging*

• Finding 1. The cyberinfrastructure environment in the US is now much more complex and varied than the long-useful Branscomb Pyramid. …This is largely due to the maturity of commercial cloud facilities, volunteer computing….

• Finding 2. The reward system as perceived by individual faculty researchers in science and engineering does not support a focus on Virtual Organizations as an essential organizational feature in scientific and engineering research. It encourages a highly diffuse, uncoordinated cyberinfrastructure that makes sharing and collective investment difficult and does not optimize the effectiveness of CI support for research and development in science and engineering in the United States.

• Finding 3. The US open science and engineering research community is not using the existing cyberinfrastructure as effectively or efficiently as possible, primarily as a result of the current state of cyberinfrastructure software and the resulting barriers of migration among and between the many and varied campus and national cyberinfrastructure facilities.

7

*From: NSF Advisory Committee for Cyberinfrastructure Task Force on Campus Bridging. Final Report. March 2011. http://www.nsf.gov/od/oci/taskforces/TaskForceReport_CampusBridging.pdf.

(In)adequacy of Research CI

Never (10.6%)

Some of the time (20.2%)

Most of the time (40.2%)

All of the time (29%)

8

From: NSF Advisory Committee for Cyberinfrastructure Task Force on Campus Bridging. Final Report. March 2011. http://www.nsf.gov/od/oci/taskforces/TaskForceReport_CampusBridging.pdf.

Not the biggest surprise this year…

• Finding 4. The existing, aggregate, national cyberinfrastructure is not adequate to meet current or future needs of the US open science and engineering research community.

• Finding 5: A healthy national cyberinfrastructure ecosystem is essential to US science and engineering research and to US global competitiveness in science and technology. Federal R&D funding overall is not sufficient to meet those needs, and the NSF share of this funding is not sufficient to meet even the needs of basic research in those disciplines that the NSF supports.

• A key point of the entire Task Force’s work is that NSF funding alone is insufficient to solve the nation’s CI problems but that NSF leadership and use of funding to align expenditures nationally (a la NSFNet) can have tremendous impact on the nation.

9

Strategic Recommendations to NSF, part 1Strategic Recommendation to the NSF #1: As part of a strategy of coherence between NSF and campus CI and reducing reimplementation of multiple authentication systems, the NSF should encourage the use of the InCommon Federation global federated system by using it in the services it deploys and supports, unless there are specific technical or risk management barriers. 10

http://www.incommon.org/cyberroadmap.html

Cyberinfrastructure is infrastructure

Strategic Recommendation to the NSF #2: NSF must lead the community in establishing a blueprint for a National CI

Specific suggestions on how to do this, rather than what to do specifically, made in report

11

Illustration from National Science Foundation. Investing in America’s Future: Strategic Plan FY 2006-2011. September 2006. Available from: http://www.nsf.gov/pubs/2006/nsf0648/nsf0648.jsp

ACCI Campus Bridging Task Force Strategic Recommendations to NSF, part 2

• Strategic Recommendation to the NSF #3: The NSF should create a new program funding high-speed (currently 10 Gbps) connections from campuses to the nearest landing point for a national network backbone. …

• Strategic Recommendation to the NSF #4: The NSF should fund national facilities for at least short-term storage and management of data to support collaboration, scientific workflows, and remote visualization; management tools should include support for provenance and metadata. …

• Strategic Recommendation to the NSF #5: The NSF should continue research, development, and delivery of new networking technologies….

• Strategic Recommendation to the NSF #6: The NSF should fund activities that support the evolution and maturation of cyberinfrastructure through careful analyses of needs (in advance of creating new CI facilities) and outcomes (during and after the use of CI facilities).. … All studies of CI needs and outcome, including ongoing studies of existing CI facilities, should be published in the open, refereed, scholarly literature.

12

ACCI Campus Bridging Task Force Strategic Recommendations to university leaders and the US higher education community

• Strategic Recommendation to university leaders and the US higher education community #1: Institutions of higher education should lead efforts to fund and invest in university-specific, state-centric, and regional cyberinfrastructure to create local benefits (in research accomplishment and local economic development) and to aid the global competitiveness of the US and thus the long-term welfare of US citizens.

• Strategic Recommendation to university leaders and the US higher education community #2: Every institution of higher education should have a plan, developed and endorsed at the highest level of its governance, for the establishment of a coherent cyberinfrastructure. …

• Strategic Recommendation to university leaders and the US higher education community #3: Institutions of higher education should adopt criteria for tenure and promotion that reward the range of contributions involved in the production of digital artifacts of scholarship. …

13

XSEDE Campus Bridging vision• Create the software and training tools that will allow excellent interoperation between

XSEDE infrastructure, and excellent usability from the researcher’s standpoint for a variety of modalities and types of computing: traditional HPC as well as data and High Throughput Computing as well

• Promote better use, via XSEDE and campus bridging tools, of the nation’s aggregate CI resources – Recommending the use of InCommon for all authentication systems– By making it easier to use contribute campus systems (in whole possibly but generally

in part) to the aggregate capacity and capability of XSEDE– By making it easier to use systems that are not contributed to the aggregate of XSEDE

overall more effectively in the context of workflows and cyberinfrastructure that include resources within and beyond XSEDE in a well coordinated fashion

• Our goal is going to be to work with the various groups in XSEDE (particularly XAUS, Campus Champions, Documentation / Training) to align activities and communications so that XSEDE collectively does things in a way that achieves the goals above

14

XSEDE Campus Bridging activities

• To be conscientiously targeted at Data, HPC, and HTC – probably in that order

• Working closely with architecture team to help disseminate XSEDE's plans– XSEDE architecture plans => out to campus champions and community– Funnel community response => back to XAUS

• Strategy: conscientiously make a small number of reasoned choices, pursue them with diligence, and reap economies of scale (if things go right) or clear learning experiences (otherwise)

15

XSEDE Campus Bridging tactics

• Tools for doing this:– Installers (thoroughly tested)– Documentation & training– Science Gateways (document by Surresh Marru and Marlon Pierce) – Ability to contribute community resources for greater good– Pilot testing program

16

XSEDE Campus Bridging staff

• Craig Stewart (reports to Scott Lathrop in Scott’s role leading outreach)

• Jim Ferguson (works 25% on campus bridging)• Therese Miller – IU overall project lead for XSEDE activities

(will be aiding, expected particularly in regards to campus champions)

• Rich Knepper – Manager, Core Services, RT/PTI (0.25 FTE starting next year)

• 1.0 FTE for “ROCKS Rolling” for PY2

17

Five-year goals

• At the end of 5 years, have implemented and socialized a community vision of an integrated national cyberinfrastructure with XSEDE as a critical component – but just a component

• Working with XSEDE and the community as a whole, implement the technology required to support as many of the use cases described here as possible

Campus Bridging use cases

• UCCB 1.0. InCommon-based Authentication. Consistent use of community-accepted authentication mechanisms.

• UCCB 2.0. Economies of scale in training and usability • UCCB 3.0. Long-term remote interactive graphic session • UCCB 4.0. Use of data resources from campus on XSEDE, or from

XSEDE at a campus • UCCB 5.0. Support for distributed workflows spanning XSEDE and

campus-based data, computational, and/or visualization resources • UCCB 6.0. Shared use of computational facilities mediated or

facilitated by XSEDE • UCCB 7.0 Access to resources on a service for money basis (___ on

demand). • CB Prerequisite. XSEDE-wide unified trouble ticket handling

More consistency in CI setups => economies of scale

20

• In reality, the four cluster admins depicted here being in agreement are all right.

• Experienced cluster admins all learned how to use what they learn when the tools were still developing, so the tool each sysadmin knows the best is the tool that lets that sysadmin do their work the best

• The only way to develop consistency is to provide installers that will make their work easier

• The XSEDE architecture group is developing installers for file management tools

• *A la Steven Colbert, the “4 out of 5…” comment is not intended to be a factual statement

Training and outreach

Image from TeraGridEOT: Education, Outreach, and Training 2010. https://www.teragrid.org/web/news/news#2010scihigh

• Consistency in system setups – local becoming more like XSEDE – should also lead to economies of scale in training

• Materials and trainer expertise will be more easily transportable and extensible

• The campus bridging group plans to work very closely with the campus champions

Campus Bridging GFFS pilot program

Texas A&M – use of GFFS with Brazos Cluster Univ of Kansas – data transfer within Great Plans and with

Indiana University CUNY – spanning campus to XSEDE resources Univ of Miami – data sharing within campus and across WAN Pilot projects are currently working with XSEDE Operations to

prepare Genesis II Infrastructure for beta usage

Next deliverable to be delivered

• Template for hybrid system description – generalization of the TACC / IU templates

• Paper at XSEDE12• Some real progress on pilots• ROCKS Rolls – mid PY2 first visible results likely

23

List of work products in addition to task force report (available from IU ScholarWorks and for larger pieces Amazon CreateSpace)

• Report on Campus Bridging Technologies Workshop: Networking and Data Centric Issues.

• Report on Campus Bridging Technologies Workshop: Campus Bridging Software and Software Service Issues. – http://hdl.handle.net/2022/13070

• Report on Campus Leadership Engagement in Building a Coherent Campus Cyberinfrastructure.

• A Roadmap for Using NSF Cyberinfrastructure with InCommon – http://www.incommonfederation.org/cyberroadmap.html

• A Roadmap for Using NSF Cyberinfrastructure with InCommon: Abbreviated Version

24

http://www.nsf.gov/od/oci/taskforces/

25

Thanks• Campus Champions – your input has already shaped Campus Bridging activities greatly.• Guy Almes, Von Welch, Patrick Dreher, Jim Pepin, Dave Jent, Stan Ahalt, Bill Barnett, Therese Miller,

Malinda Lingwall, Maria Morris• Gabrielle Allen, Jennifer Schopf, Ed Seidel, all of the NSF program officers involved. (Anyone who is happy

that there are less than 87 recommendations owe Jennifer a special thanks ; I owe Gabrielle special thanks for special effort on publication matters generally)

• All of the IU Research Technologies and Pervasive Technology Institute staff who have contributed to this entire 2+ year process

• Special thanks to CASC members who have participated in one of n information gathering exercises (where n is large)

• NSF for funding support (Awards 040777, 1059812, 0948142, 1002526, 0829462)• Lilly Endowment and the Indiana University Pervasive Technology Institute• Any opinions presented here are those of the presenter or collective opinions of members of the Task

Force on Campus Bridging and do not necessarily represent the opinions of the National Science Foundation or any other funding agencies

• Tevfik Kosar, who as Chair of DIDC ‘10 invited me to present the t Keynote presentation at the Third International Workshop on Data Intensive Distributed Computing (DIDC'10) held in conjunction with HPDC'10, Chicago IL. “It’s not a data deluge – it’s worse than that.” Several slides from that talk are reused here. That original talk is available online (Creative Commons 3.0 Unported license) Location: http://hdl.handle.net/2022/13195

• CAMPUS CHAMPIONS – WHO ARE KEY PARTNERS IN CAMPUS BRIDGING ON XSEDE!

26