CCSM Software EngineeringMariana Vertenstein

CCSM Software Engineering Group (CSEG)

CCSM SE Goals

• To permit new scientific experiments to be seamlessly carried out using CCSM • CCSM software engineering is there to serve the scientific goals of climate modeling and SE “best practices” might not always directly apply

CCSM “Hub and Spoke” System

cpl

atm

ocn ice

lnd

CCSM consists of a system of geophysical components that CCSM consists of a system of geophysical components that exchange information via a flux coupler exchange information via a flux coupler

CCSM Overview• Coupler features:

• Communicates with all components• Only exchanges 2d states/fluxes• Maps/interpolates data• Merges fields and communicates some fluxe

• Component features:• Can be “active” or “data”

• Active components are CAM, CLM, CSIM, POP• Data components are DATM, DLND, DICE, DOCN

• Each component is a separate executable and runs on a unique set of processors

• Entire system is an multiple executable system where each component runs concurrently

• Single executable version is on the way

CCSM Overview (cont)

• CCSM has plug and play functionality (active and data components can be swapped at compile time).

• CCSM active components can also run as stand-alone binaries with "proprietary" forcing normally provided by CCSM coupler.

• CCSM currently duplicates most stand-alone functionality. • normally involves a sizeable performance penalty

• maintaining stand-alone functionality separate from CCSM results in additional software costs

CCSM3 Accomplishments

IPCC AR4 CCSM3.0 was main contributor to IPCC AR4

9.6 Tbytes of data produced (25% of holdings at PCMDI for IPCC AR4)

9100 years of T85 simulationsCCSM3.0

Released in June 2004 450 downloads since release was put on ESG (~1 download/day)

Input and output data available on ESG

CCSM Management

At top level, CCSM consists of a collection of 10 working groups. The CCSM Working Groups are groups of scientists that work on individual component models or specific coupling strategies.

Each team takes responsibility for developing and continually improving its component of the CCSM.

Each team will decide their own development priorities and work schedules, consistent with the overall goals of CCSM, and subject to oversight by the CCSM Scientific Steering Committee (SSC).

CCSM Management (cont)

• SEWG created in 2001 - now also has representation on the SSC

• CSEG - part of SEWG• Created 5 years ago - before then there was no cohesive software engineering structure to CCSM

• Consists of 9 SE’s• Responsible for all software engineering aspects of the entire CCSM system

• Have our own internal web pages and weekly meetings

CCSM SE Challenges

• CCSM is a community model - not an NCAR model. We work with a wide variety of collaborators inside and outside of NCAR that bring new science and software into the CCSM system - do not develop all of our software “in house” or even under NCAR control

• New target platforms and architectures are constantly being added.

• Code re-factoring and incorporation of new science must occur simultaneously. Example - API's were not part of legacy code base e.g. input/output arguments via "use" statements often found throughout code.

CCSM SE Challenges (cont)

• Significant code duplication currently exists • CAM has stand-alone slab ocean functionality

• CSIM has stand-alone slab ocean functionality

• CCSM/DOCN has its own slab ocean component• Stand-alone component builds are often different from CCSM build system.

• New software capability and science usually are incorporated and tested in stand-alone component versions.• These are coupled to CCSM system as final step and often result in unexpected behavior (e.g. CAM finite volume dynamical core in CCSM)

CCSM SE Challenges (cont)

• Model experiments must be able to run on a wide variety of target platforms. For each target platform: • Experimental throughput (model years/wall clock day) must be optimized

• Extensive testing and climate validation of the code base must be performed.

• Introduction of new science and new platforms often results in new software requirements. As examples:• New science might have potentially large impact on memory requirements.

• Vector platforms often introduce unexpected behavior in the system.

• Code base will have to address future scalability issues for new architectures down the road.

CCSM SE Challenges (cont)• Project management is a constant challenge.

Each component has software liaison(s) responsible for SE development related to that component• Light weight project management is only sane solution since most projects only involve one person

• Community support with limited resources is a constant challenge. • Must maintain both release version of ccsm3.0 for new platforms and development version for new science and platforms

• Legacy approach was to quickly "code and fix” - we are moving away from this

• Recent loss of resources has posed a challenge

CCSM “Best Practices”

• Revision control• Testing• Bug/Feature Tracking• Introduction of standardized interfaces (API’s)

• Code Reviews• Light weight project tracking• Light weight requirements• Communication

CCSM Revision Control• Migrated the entire CCSM code base from CVS to

Subversion this January.• Subversion has solved many of the risks and problems that existed with CVS.

• Only head of component trunks were migrated to Subversion - all previous history was kept in CVS.

• CCSM is a collection of component "tags”.• There is no independent CCSM trunk or branch.

• Each component can proceed with independent development without impacting the CCSM system.

• CCSM web tag page describes each tag and associated testing.

CCSM Revision Control (cont)

• Components have trunk and branch development lines.

• Tight control is kept over code that is committed to the trunk.

• Developers are given free rein over branch code development.

• Gatekeepers exist for component specific trunk tags and CCSM model tags.

• Any new code that changes answers must have climate simulations performed before it can be committed to the trunk.

CCSM Testing

• All testing is done with in-house software.• Currently only system regression testing is done.

• Some components (CAM, CLM) have stand-alone component test harnesses (CAM's new test harness is the poster child for CCSM testing).

• Some component testing is done automatically (post-tag in CAM) - would like to extend this in all of CCSM.

• Some components (CAM,CLM) also require that answers are independent of number of tasks/treads in MPI/OpenMP parallelization.

CCSM Testing (cont)

• CCSM has a system wide test harness that exercises different component combinations and component modes (e.g. turning on co2 exchange between the land and atmosphere).

• Future testing goals:• Upgrade the CCSM3.0 test suite to incorporate the rapidly growing parameter space (new active components, new biogeochemistry, atmospheric chemistry).

• Incorporate extensive unit tests throughout the entire CCSM system (collaborating with SciDAC to do this).

CCSM Bug Tracking

• Started using Bugzilla extensively across all components this January - fairly small time investment to start up with - large and immediate payback.

• Trying to use Bugzilla for both bug-tracking and feature tracking

• Goal is to review all outstanding component/model system bugs on a bi-weekly basis• This makes bugs more visible to all CSEG members and also makes person responsible for bug more accountable

• Migrate to newer Bugzilla interface as soon as possible.

Introduction of standardized interfaces

(API’s)• Legacy code base has often made it difficult to put changes required for new science into some components.

• Introduction of standardized interfaces will greatly simplify the addition of new science.

• Introduction of API's as part of code re-factoring has been a step forward. As an example from CAM:• New CAM physics/dynamics interfaces

Introduction of API’s (cont)

• New CAM-(CAM-surface-component) interfaces• Part of effort to construct a single executable sequential CCSM

• Introduced top layer application driver and thin coupling layer to permit different coupling frameworks to be evaluated

• Surface components can then be replaced by standard CCSM components (e.g. CAM/CAM-OCN -> CAM/DOCN)

CCSM Code Reviews

• New code that goes on component trunks is normally required to undergo a code review.

• Normally the component gatekeeper is part of the code review process.

• Code reviews have been extremely helpful in flushing out problems before they are committed to the repository.

• Code review is often only done before revision control check-ins - more frequent code reviews are a goal.

CCSM Project Tracking

• Adopting swikis for project tracking.• swiki is a collaborative work area and web page that is easily modifiable by its users

• For CSEG - light weight project tracking using NCAR swiki has proved very successful.• Each “project” has its own swiki• Requirements, design documents, timelines and tasks are all implemented in the swiki

• Scientists are also adopting the swiki • Better tools are on the horizon.

CCSM Light Weight Requirements

• Formal requirements have not worked - the time input required to establish more formal requirements has not paid off.

• CCSM normally uses requirements as a design tool.

• Projects are normally single-person efforts and as such are best suited to light weight requirements that do not require a significant up-front time commitment to do.

• Where we need to improve:• Requirements are often not looked at once they have been written down

• Requirements change as the code design and implementation occur - but requirements document is usually not modified to keep up with these changes

Communication

• Introduction of bulletin board • Our only means of very limited user support - encourage community to share ideas.

• Internal part of bulletin board provides effective means of internal communication.

Role of SEA for CCSM

SEA will hopefully provide CSEG with a much needed venue for exchanging information with other NCAR software engineers and bringing in

outside expertise that we can apply to meet our specific SE

goals.