Experiences Developing Tools for Scientific Communities in the Apache Software Foundation:

Beyond Open Source

Marlon PierceIndiana UniversityFebruary 14, 2012

IU Science Gateway Group Overview• IU Science Gateway Group members

– Marlon Pierce: Group Lead– Suresh Marru: Principal Software Architect– Raminder Singh, Chathura Herath, Yu Ma, Lahiru Gunathilake:

Senior team members– Five interns and research assistants

• NSF SDCI funding of Open Gateway Computing Environments project – TACC (M. Dahan), SDSC (N. Wilkins-Diehr), SDSU (M. Thomas),

NCSA (S. Pamidighantam), UIUC (S. Wang), Purdue (C. Song), UTHSCSA (E. Brookes)

• We participate in two Apache incubators– Apache Rave: http://incubator.apache.org/rave/– Apache Airavata: http://www.airavata.org

Key Problems for Science Gateway, CI Software

• Reusability– Reuse or write own gateway software?

• Sustainability– Small teams– Successful gateways become infrastructure

• Hard to fund– Cyberinfrastructure Software Sustainability and Reusability:

Report from an NSF-funded workshop• https://scholarworks.iu.edu/dspace/handle/2022/6701

• Governance– How are design decisions made?– Who decides if the software is suitable for release?– How do you handle contributions?– How do you add people to the development and project

management teams?

Proposed Solution: Open Community Software

• More than SourceForge, GitHub, Google Code, etc– Those provide excellent Web tools to help

developers.• Here we are concerned with community

building.– Diverse community of developers means

reusability and sustainability– But diverse communities require governance

Some Open Model Examples in CI• NSF-funded CDIGS project

– http://confluence.globus.org/display/CDIGS/CDIGS+Home+Page• HUBzero Consortium, Sakai Foundation

– Institutional level organizations • Eclipse Parallel Tools Platform

– Jay Alameda, NCSA: NSF SI2 funding to develop HPC tools workbench

– http://www.eclipse.org/ptp/• Enzo Project

– Excellent talk at TG11 by Prof. Brian O’Shea on their open community efforts

– http://enzo-project.org/• Apache Software Foundation

– OODT and TIKKA Data Management Projects at NASA JPL

Two Apache Software Foundation Case Studies

Apache Rave and Airavata Incubators

Apache Airavata

• Science Gateway software framework to– Compose, manage, execute, and monitor

computational workflows on Grids and Clouds– Web service abstractions to legacy command line-

driven scientific applications– Modular software framework to use as individual

components or as an integrated solution.• More Information

– Airavata Web Site: airavata.org– Developer Mailing Lists: airavata-

dev@incubator.apache.org

Apache Airavata High Level Overview

Example Workflow: Nuclear Physics

Courtesy of collaboration with Prof. James Vary and team, Iowa State

1

Apache Rave

• Open Community Software for Enterprise Social Networking, Shareable Web Components, and Science Gateways

• Founding members:• Mitre Software• SURFnet• Hippo Software• Indiana University

• More information• Project Website: http://incubator.apache.org/rave/• Mailing List: rave-dev@incubator.apache.org

Gadget Dashboard View

Gadget Store View

Extending Rave for Science Gateways

• Rave is designed to be extended.– Good design (interfaces, easily pluggable

implementations) and code organization are required.– It helps to have a diverse, distributed developer

community• How can you work on it if we can’t work on it?

• Rave is also packaged so that you can extend it without touching the source tree.

• GCE11 paper presented 3 case studies for Science Gateways

The Apache Way and Science Gateways

Why Apache for Gateway Software?• Apache Software Foundation is a neutral playing field

– 501(c)(3) non-profit organization.– Designed to encourage competitors to collaborate on

foundational software.– Includes a legal cell for legal issues.

• Foundation itself is sustainable– Incorporated in 1999– Multiple sponsors (Yahoo, Microsoft, Google, AMD, Facebook,

IBM, …)• Proven governance models

– Projects are run by Program Management Committees.– New projects must go through incubation.

• Provides the social infrastructure for building communities.• Opportunities to collaborate with other Apache projects

outside the usual CI world.

The Apache Way• Projects start as incubators with 1 champion and several mentors.

– Making good choices is very important• Graduation ultimately is judged by the Apache community.

– +1/-1 votes on the incubator list• Good, open engineering practices required

– DEV mailing list design discussions, issue tracking– Jira contributions– Important decisions are voted on

• Properly packaged code– Build out of the box– Releases are signed– Licenses, disclaimers, notices, change logs, etc.– Releases are voted

• Developer diversity– Three or more unconnected developers– Price is giving up sole ownership, replace with meritocracy

Apache and Science Gateways

• Apache rewards projects for cross-pollination.– Connecting with complementary Apache projects

strengthens both sides.– New requirements, new development methods

• Apache methods foster sustainability– Building communities of developers, not just users– Key merit criterion

• Apache methods provide governance– Incubators learn best practices from mentors– Releases are peer-reviewed

Apache Contributions Aren’t Just Software

• Apache committers and PMC members aren’t just code writers.

• Successful communities also include– Important users– Project evangelists – Content providers: documentation, tutorials– Testers, requirements providers, and constructive

complainers • Using Jira and mailing lists

– Anything else that needs doing.

How To Get Involved

• Join the DEV mailing lists.• Grab the software and start complaining.• Post Jira tickets

– Add your patches to Jira if you want to solve a problem.

– Frequent patch submission is the best way to get voted in as a committer.