Collaborative Software Development inSupport of Fast Adaptive AeroSpace Tools

(FAAST)

William L. Kleb∗, Eric J. Nielsen†, Peter A. Gnoffo‡, Michael A. Park†, and William A. Wood∗

NASA Langley Research Center, Hampton, Virginia 23681

Corresponding author: Bil.Kleb@NASA.Govphone: +1 757 864 4364, fax: +1 757 864 8670

A collaborative software development approach is described. The softwareproduct is an adaptation of proven computational capabilities combined with new ca-pabilities to form the Agency’s next generation aerothermodynamic and aerodynamicanalysis and design tools. To efficiently produce a cohesive, robust, and extensiblesoftware suite, the approach uses agile software development techniques; specifically,project retrospectives, the Scrum status meeting format, and a subset of ExtremeProgramming’s coding practices are employed. Examples are provided which demon-strate the substantial benefits derived from employing these practices. Also includedis a discussion of issues encountered when porting legacy Fortran 77 code to For-tran 95 and a Fortran 95 coding standard.

IntroductionThe objective of the Fast Adaptive AeroSpace

Tools (FAAST) program at NASA Langley ResearchCenter is to develop the next generation of aerospaceanalysis and design tools.1 The four primary ele-ments in this effort are CAD-to-Grid Methods, HighEnergy Flow Solver Synthesis (HEFSS), OptimallyConvergent Algorithms, and Efficient Adjoint De-sign Methods. This paper primarily focuses onthe software development practices adopted by theHEFSS and design elements of FAAST.

Over the past two decades, Langley’s Aerother-modynamics Branch has provided extensive compu-tational support for NASA’s space program. Con-tributions have included aerodynamic and aerother-modynamic predictions across the hypersonic regimefor planetary missions such as Mars Pathfinder andaccess to space projects such as X-33, X-34, and∗Aerospace Engineer, Aerothermodynamics Branch, Aero-

dynamics, Aerothermodynamics, & Acoustics Competency,member AIAA.†Aerospace Engineer, Computational Methods Branch,

Aerodynamics, Aerothermodynamics, & Acoustics Compe-tency, member AIAA.‡Aerospace Engineer, Aerothermodynamics Branch, Aero-

dynamics, Aerothermodynamics, & Acoustics Competency,AIAA fellow.

The use of trademarks or names of manufacturers in thisreport is for accurate reporting and does not constitute an officalendorsement, either expressed or implied, of such products ormanufacturers.

This paper is a work of the U.S. Government and is not subjectto copyright protection in the United States.

X-37. The primary tool used to provide these datahas been the Laura solver,2 developed by Gnoffo.In addition to external hypersonic flows, Lang-ley’s Hypersonic Airbreathing Propulsion Branchhas made significant contributions to NASA’s hyper-sonic propulsion projects with the Vulcan solver,3

developed by White. Together, these two packagesrepresent the state of the art at Langley in high-energy, reacting-gas chemistry computational tools.

While success has been achieved using the struc-tured grid tools Laura and Vulcan, there areinherent topology limitations on geometric config-urations for which structured-grid discrete domainscan be efficiently produced. Alternatively, the fieldof unstructured-grid methods has matured rapidly inrecent years. With this approach, a wide range of ge-ometric configurations can be efficiently modeled foranalysis. Langley tools such as Fun2D/3D4,5 andUsm3D6 have been validated with experimental dataand with their structured-grid counterparts for awide array of perfect-gas Reynolds-averaged NavierStokes applications, ranging from incompressible totransonic and supersonic flows.

The goal of the HEFSS project is to com-bine the capabilities and strengths of the reacting-gas physical models in Laura and Vulcan andthe unstructured-grid discretizations of codes likeFun3D and Usm3D to produce the next-generationcomputational tool for analysis and design, whileemploying software development techniques that en-

1 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

able a robust, extensible, and portable final prod-uct. This software development effort is similar toONERA’s elsA project.7

The development of computational fluid dynamic(CFD) application codesa at Langley and at the for-mer, co-located Institute for Computer Applicationsin Science and Engineering (ICASE) has consisted ofone or two people working sharply focused applica-tions or algorithms. Even if more people contributeto the development of a code, there is typically onlyone person who contributes the bulk of the codeand serves as gatekeeper for any changes. Mani-festations of this paradigm are shown in Table 1. Inall these examples, the code architect will cite oth-ers who have made important contributions to thecode; nevertheless, they are typically managed likea cottage industry.

In cases where the high-level application is identi-cal, algorithm details may differ because of code ro-bustness considerations for specialized applicationsor because new algorithms must eventually evolveto “application” status. Such evolution has oftenbeen easier to accommodate by extending the ini-tial, simple test versions of the algorithm ratherthan by integrating them into an existing applicationcode. However, over the last decade, both the fieldof CFD and computational capability have largelyoutstripped the ability of a single developer to makea significant contribution. The scope and complexityof a modern application now require several expertsto work collaboratively.

Software engineering processes which accommo-date teams of tens of hundreds of programmersworking with a relatively well-defined set of require-ments (e.g., a satellite tracking system) were con-sidered,b but it was found that they are simplynot appropriate for a small, research environment.On the contrary, the emerging agile software devel-opment movementc is well suited to the uncertainrequirements and small teams typically present ina research environment. The agile movement viewssoftware development as an empirical process ratherthan the defined process which software engineer-ing attempts to govern.8 To manage the empiri-cal process, agile methods incorporate rapid feed-back mechanisms to enable constant steering andplace a renewed emphasis on the heart of softwaredevelopment—software craftsmanship.9

aAn “application” code is defined as one which can com-pute aerodynamics and/or aerothermodynamics of three-dimensional flows, including appropriate physical models andboundary conditions for geometrically complex configura-tions.

bSee www.sei.cmu.edu/cmm/ for example.cSee www.agilealliance.org.

Regardless of the software development processchosen, making the switch from a one-code, one-developer paradigm to a team-based approach is alarge culture change under any circumstances. How-ever, the ambitious goals of the HEFSS project pro-vided a strong motivation to look past skepticismand overcome resistance to change since it requireda group of developers (initially 12 people rangingbetween 25 to 100 percent work-level), with diverseareas of expertise, to collaborate on a single pieceof software. Within the 18 months of the project,HEFSS had promised to demonstrate successful syn-thesis of the structured-grid physical models on acylinder case using an unstructured discretization.In addition, an existing unstructured-grid code wasto be selected to serve as the baseline for the HEFSSeffort, and its functionality was to be maintainedwithin the HEFSS code base. To compound mat-ters, there were no software development expertsavailable to serve on the team. This critical gap wasfilled by consultant-led workshops, a visiting lecturerseries, a support contractor, and the good fortuneof having two team members with CFD expertiseaggressively pursue software development best prac-tices appropriate for our team.10,11

The purpose of this paper is to document howthe HEFSS team adapted and incorporated agilesoftware development practices to develop the nextgeneration CFD application software. No claims aremade that the correct process decisions were made orthat the current processes have fully matured. Andsince there is no control team with identical talentsand objectives, it is difficult to objectively gauge theperformance of the HEFSS team other than to ob-serve that the project is ongoing, morale is high, itspractices have been adopted by other teams, it wasincluded in a group achievement award, and the lo-cal software engineering process group is using it asa model. The experience and lessons learned arehumbly offered as a case study, which may be usefulto others with similar background and goals.

The first several sections outline the baseline codeselection process, justify the programming languagechosen, and outline how the legacy code base wasported and restructured to take advantage of thenew language features. Following this backgroundmaterial, modularity and data encapsulation designissues are presented as well as details about how thehigh-energy physics modules were incorporated intothe baseline solver. Next, the software developmentsection documents the practices that allow a teamsoftware development environment to thrive. Thisdiscussion is followed by a section highlighting sev-eral experience reports of research products created

2 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Table 1 CFD code, architect, and application domain.

Cfl3D Rumsey/Biedron Structured-grid (SG) aerodynamics

Laura Gnoffo SG hypersonic aerothermodynamics

Vulcan White SG hypersonic propulsion

Tlns3D Vatsa SG aerodynamics

Overflow Buning Overset SG aerodynamics

Usm3D Frink Unstructured-grid (UG) aerodynamics

Nsu3D Mavriplis UG aerodynamics

Fun3D Anderson UG aerodynamics and design

Felisa Peraire UG hypersonic aerodynamics

in this environment and finally, some concluding re-marks.

Baseline Code SelectionThree Langley unstructured codes, Usm3D,

Fun3D, and Felisa,12 were considered as the initialtemplate for the Hefss code. Felisa, an inviscid,unstructured flow solver, already has considerablesuccess in the hypersonic domain. It also has equilib-rium and thermochemical nonequilibrium gas mod-els. While the addition of thermochemical nonequi-librium source terms, thermodynamic models, andtransport models was perceived to be straightfor-ward, considerable effort would have been requiredto introduce the viscous terms, the viscous flux Ja-cobians, and an implicit solution scheme. BothUsm3D and Fun3D are highly successful codes forcomputing viscous flow on unstructured grids withinthe subsonic to low supersonic speed regimes. Ul-timately, Fun3D was selected because it is morerobust in the hypersonic domain, which is appar-ently attributable to its combination of Roe FluxDifference Splitting, flux reconstruction, and asso-ciated limiters. In addition, its discretizations aresimilar to Laura, and the discrete adjoint capabil-ity for perfect gas design13,14,15 and grid adapta-tion16,17,18,19,20,21 was judged particularly appeal-ing for future hypersonic design and grid adaptation.A successful retrofitting of Fun2D with thermo-chemical nonequilibrium models confirmed the vi-ability of this approach.

Programming LanguageMost of the CFD codes developed at Langley

are written in Fortran 77 and often rely on non-portable extensions such as vendor-specific functionsor links with C code. For the current project, theteam sought a single, unifying standard language un-der which to develop new code. After surveying theavailable programming languages and deciding thata mixed-language code base would increase complex-ity too much, Fortran 95 was selected for the newsuite of codes. Fortran 95 promises the numerical

performance of Fortran 77 with the advanced fea-tures of other languages, such as dynamic memoryallocation, derived types, recursion, and modules.This choice also allows a relatively straightforwardconversion of a substantial legacy code base writtenin Fortran 77.

The selection of Fortran 95 was tempered by thecommitment to deliver a hypersonic flow simulationwith thermochemical nonequilibrium on a geometri-cally simple configuration within 18 months. Adop-tion of a programming language significantly differ-ent from Fortran would have required a learningperiod for the majority of the team members, whowere already proficient with Fortran 77. The timerequired to bring team members up to speed in a newlanguage, plus the time required for conversion oflegacy Fortran 77 to a language outside the For-

tran family, was judged too costly, relative to thepotential benefit offered by any other language.

Fortran 95 training was tailored to team needsin a two-part workshop. Dan Nagle, from PurpleSage Computing Solutions,d spent a day with theteam learning the Hefss code objectives and thearchitecture of the legacy code. Using this mate-rial, he prepared a two-day course which highlightedFortran 95 features suited to the HEFSS project.

Auxiliary scripting for controlling code compi-lation, templating, and testing is performed withRuby22,23 and Make.e Ruby is an open source,object-oriented, threaded-scripting language withcross-platform support, while Make is an opensource compilation tool.

Porting and RestructuringLegacy Code

To lay a solid foundation for the new suiteof solvers, Fun3D and the physical models fromLaura and Vulcan were ported from a mixture ofC and Fortran 77 to Fortran 95. Porting For-

tran 77 code to Fortran 95 was initially thought

dusers.erols.com/dnagle/ewww.gnu.org/software/make/make.html

3 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

to be a simple process that could be accommo-dated by using a combination of homegrown scriptsand a commercial software package, Foresys

�.f

Foresys� was helpful when implicit none was

requested because it would automatically declareall variables used in the routine. It also providedinstructive diagnostics for various classes of errorsduring the conversion process and when replacingcommon blocks by modules. However, it invariablyreformatted lines and destroyed symmetric formsof equations that had been carefully introduced byearlier authors, and it repositioned or silently elimi-nated comments. Eventually, Ruby and Perl scriptswere crafted to handle tedious, error-prone opera-tions such as code indentation and the conversion ofcontinuation symbols without losing the commentsand other structured formatting. The remainder ofthe conversion was done manually.

As the team had a chance to study the legacystructure, it became clear that the old arrangementsof common blocks and subroutines were counter tothe modularity and extensibility the team was try-ing to create. So, during the port to Fortran 95,common routines and functions were extracted andplaced in a single, shared library directory, whiledata structures such as boundary conditions, gridmetrics, and solution quantities were generalized tohandle an arbitrary number of equations and wereencapsulated in derived types.

The use of derived types provides additional flex-ibility over Fortran 77; however, early versionsof Fortran 95 compilers often displayed a signif-icant performance penalty when these constructswere used in the computationally intensive regionsof the solver.g Consequently, the restructuring ef-fort often required reworking these core routines torecover performance comparable to the legacy solver.

This transformation took nearly a year and wasnot without difficulties, but it was definitely a worth-while effort because it gave team members hands-onexperience with a code most had never seen before,instead of merely accepting the results of an auto-matic conversion. The conversion process also gavethe team an opportunity to create and tailor a cod-ing standardh suited to their style and knowledge.In addition, the total lines of source code had beenreduced by some 40 percent, in itself a significantbenefit from the standpoint of code maintenance.

fForesys

� is a trademark of Connexite S.A., for moreinformation see www.simulog.fr/is/2fore1.htm.

gSee Appendix B on page 17 for current results.hSee Appendix A on page 16.

Modularity and Encapsulation

Modularization, along with abstraction, informa-tion hiding, and encapsulation, are also means usedto enhance code maintainability and bring the ad-ditional promises of code reuse, reduced complexity,extensibility, and orthogonality.i Abstraction is theprocess of picking out common features of objects orprocedures and replacing them with a single, moregeneral function. Information hiding reduces com-plexity by hiding details of an object or functionso the developer can focus on the object withoutworry about the hidden details. Encapsulation, orcombining elements to create a larger entity, is onemechanism to achieve this.

The Fortran 95 constructs of modules, inter-face statements, public and private declarations, andderived types were employed to implement theseideas. Fortran 95 modules are similar to theclass construct in object-oriented languages, whilederived types are akin to structures. Moduleswere designed to abstract types of operations, e.g.,file input/output, memory allocation, interproces-sor communication, execution timing, linear algebra,and so on. Many modules employ a generic inter-face statement that automatically detects the type,kind, and rank of the calling arguments at compiletime and matches them to an appropriate low-levelroutine, which allows them to be largely indepen-dent of any particular flow solver since data is onlyexchanged through well-defined interfaces. Manyof these Fortran 95 interface statements are pro-duced automatically in the build process by a Rubyscript which emulates the template system availablein C++. In the remainder of this section, specificexamples are given to demonstrate the benefits ofmodularization and data encapsulation.

Memory allocation

Array memory allocation is handled by a singleinterface statement in a module that automaticallydetects the type, kind, and rank of the argument andcalls the appropriate low-level routine for the alloca-tion and initialization. This abstraction streamlinesmemory allocation requests throughout the codesince memory tracking and diagnostics can be placedand maintained in a single location.

Parallel Communication

Originally, the baseline solver relied on a shared-memory implementation specific to SGI® hardwareand was not portable to the increasingly popularcluster-based, distributed-memory computing plat-

iIn this case orthogonal is used in the sense of mutuallyindependent or well separated.

4 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

forms. Moreover, the communication operationswere dispersed throughout the solver, and any mod-ifications to the communication model needed to bemade in numerous locations throughout the code.In the current work, the message passing interface(MPI) standard was selected. Interprocessor com-munication has been abstracted from all but thelowest levels of the source code and is now encap-sulated in a single module.

With this centralized approach to MPI communi-cation, it is now trivial to make sweeping changesto the parallel aspects of the code, including com-pletely removing it to produce a sequential versionof the code. This abstraction also benefited theteam when the high-energy, reacting-gas portion ofthe code was parallelized successfully on the first at-tempt. Normally, a developer would expect to spendconsiderable time debugging interprocessor commu-nication.

Boundary Conditions

Another area in which modularity and data en-capsulation have provided a significant benefit is inthe treatment of boundary conditions. The baselineFun3D solver was extremely deficient in its abilityto handle a wide range of boundary conditions. Theuser was restricted to inviscid, inflow/outflow, andviscous boundary types. Information required forthese boundary types was contained in hard-codeddata structures specific to each condition and weredispersed throughout the code. This design had be-come extremely limiting in recent applications andwas clearly not sufficient for extension to high-energyflows, where a large array of boundary conditiontypes are required.

Using Fortran 95 derived types to encapsulateboundary condition information, the baseline solverwas completely refactored to allow the straightfor-ward addition of new boundary types. For any givenboundary condition, all necessary data are containedin a boundary condition type. An array of these de-rived types then constitutes all boundaries in a givenproblem. For boundary conditions requiring addi-tional physical data, a link to an additional datastructure specific to that boundary condition is en-capsulated. Derived types also allow the additionalenrichment of the data structure without modifyingargument lists. In this manner, any number of dif-ferent boundary groups can be efficiently handled atthe higher levels of the solver and unrolled for useas needed.

It should be noted that this data structure alsoallows for a natural handling of cost functions basedon boundary data required for the design and grid

adaptation capabilities within FAAST. Objectivefunctions composed of viscous and/or pressure con-tributions can easily be specified on any subset orcombination of boundary groups such that a specificflow feature or region of the domain can be targeted.For example, if it is determined that a strong shockon the outboard section of a wing is responsible fora severe wave drag penalty, a cost function can eas-ily be formulated based solely on the contributionof that boundary group to the total drag. Thismethod represents a substantial improvement overthe baseline capabilities, where all boundary groupsnecessarily contributed to a given cost function.

Gas Physics

Modules, interfaces, and derived types are usedextensively for the gas phase physics modules, whichinclude thermodynamics, transport properties, ther-mal relaxation, and chemical kinetics. The thermo-dynamics module contains the initial interface fromthe flow solver to gas phase physics. The transportproperty module interfaces with the flow solver andthe thermodynamics module to define molecular vis-cosity, conductivity, and species diffusivities. Thethermal relaxation module is engaged when popula-tions of excited states (rotational, vibrational, andelectronic modes) cannot be defined by a single tem-perature. This module provides the source termsthat define energy exchange among the available,thermally distinct modes. The chemical kineticsmodule provides source terms for the species con-tinuity equations that define the rate of productionor destruction of species.

In conclusion, it should also be noted that becausethe HEFSS project started with a large legacy codebase and modularity and data encapsulation are elu-sive goals, which are really only earned throughexperience, code architecture changes are ongoing.In addition, there are drawbacks to modularizationthat must be considered. For example, it was origi-nally anticipated that compilers could optimize high-level constructs like derived types as if they werewritten using their lower-level counterparts. How-ever, as Appendix B on page 17 reveals, such is notalways the case in practice.

Collaborative Software DevelopmentCFD software development at Langley has tradi-

tionally been performed in a rather unconstrained,self-governed environment. As mentioned earlier,most codes have typically been developed by one, orperhaps two researchers. This paradigm has workedrelatively well and has produced software packageswidely used by industry and academia.

5 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Unfortunately, such software development strate-gies often result in codes that are complex andburdensome to maintain, and frequently subsequentworking groups produce distinct versions of the codewhich are often incompatible with each other andpreviously released versions. Moreover, cohesivenessand portability are typically lost, as additional re-searchers contribute to the code, using their owncoding style and practices.

In contrast to this ad hoc approach to code de-velopment, the HEFSS team sought to incorporatethe software industry’s best practices, not only be-cause of the challenges of working as a cohesive team,but also to find methods which would extend thelife cycle of the new code. Everyone on the teamhad experienced the pain of adding new capabilityto a large, existing code which was developed in anad hoc manner. Even a seemingly innocuous bugfix was unnerving because there was no repeatablemethod to discover whether the fix would break ex-isting capability in some subtle manner.

A survey of industry best practices for softwaredevelopment was conducted, which included spon-soring a local ICASE lecture series entitled “Mod-ern Programming Practices.”j Meanwhile, twopathfinder projects were conducted to gain hands-onexperience. Detailed discussion and extensive refer-ence lists are available in References 10 and 11.

As described earlier, the emerging body of ag-ile software development methodologies were deter-mined to have the best fit with the inconstant natureof a scientific research environment. Specifically, Ex-treme Programming (XP)24 appeared to be the mostmature, although at the time, documentation waslimited to a few websites.k In addition, recent ex-perience with ISO 9001 edicts tended to steer theteam away from defined process management tech-niques implicit in methodologies like the CapabilityMaturity Model®25 and its associated Team Soft-ware Process�.26

The collection of collaborative software develop-ment practices described herein evolved from weeklymeetings in which the challenges and possible solu-tions were discussed. Issues discussed cover fresh-start versus retro-fit versus restructuring of existingcode; language selection; coding standards; mod-ularization and maintainability versus efficiency;acceptance testing; source code management eti-quette;l and documentation. As the HEFSS teaminitially struggled with and then embraced new soft-

jSee www.icase.edu/series/MPP/kwww.c2.com and www.extremeprogramming.org.lSource code management etiquette—when source code

should and may be committed to a common repository

ware development practices, other teams (CAD-to-Grid, Design Optimization) within the FAASTproject adopted many of the same practices.

Specific software development techniques are dis-cussed in the following sections, namely: XP, projectretrospectives, status meetings, other communica-tion mechanisms, and documentation.

Extreme Programming

XP is founded on four values: communication,simplicity, feedback, and courage. It was designed tokeep the right communications flowing by employingmany practices that cannot be done without commu-nicating. XP also gambles that it is better to do asimple thing today and pay a little more tomorrowfor any necessary changes than to do a more compli-cated thing today that may never be used; that is,in this universe one cannot “save time.” Meanwhile,XP’s feedback mechanisms cover many time scalessince optimism is an occupational hazard of pro-gramming and feedback is the treatment. Finally,courage enables one to escape local optima.

Built from this value system, XP consists of 12practices shown in Table 2 on the next page. Alsoshown in the table is the level to that the HEFSSteam has adopted each practice. The ensuing sec-tions serve to briefly describe each practice and alsoto describe a practice in the context of the HEFSSteam. Adjacent to the start of each section are quo-tations from Reference 24.

Sustainable Pace Productivitydoes notincrease withhoursworked; tiredprogrammersare lessproductivethanwell-restedones.

Formally known as “40-hour week,” the sustain-able pace practice probably ranks the highest onthe common sense scale, but it is also the most fre-quently violated by managers and developers alike.Since the majority of the research conducted withthe HEFSS project is years from commercial use,compulsory overtime is simply not part of the work-ing environment.

Metaphor Guide alldevelopmentwith a simpleshared storyof how thewhole systemworks.

Employing a system metaphor which all par-ticipants can understand facilitates communicationboth within the code and within the team. Sinceall the team members are familiar with CFD jargon,the naive metaphor is used.

Coding Standards Programmerswrite all codein accordancewith rulesemphasizingcommunica-tion throughthe code.

Coding standards are usually dreaded and metwith resistance because they are seen as adding asuperfluous burden. After a brief discussion of thegenesis of HEFSS’s coding standard, several reasonsare provided to demonstrate why a coding standardis not only necessary but actually quite beneficial for

6 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Table 2 Current level of XP adoption.

Practice Adoption Comments

Sustainable pace Full No compulsory overtime.

Metaphor Full Using naive metaphor, i.e., CFD jargon.

Coding standards Full See Appendix A on page 16.

Collective ownership Full Anyone can change any piece of code.

Continuous integration Full Automated build and test on three computer architectures.

Small releases Partial A portion of code base is currently export restricted; seeking torelieve this constraint.

Test-driven development Partial Fortran 90 unit test framework not widely used; however,Ruby codes are typically created using TDD.

Refactoring Partial Performed, but not mercilessly, due to lack of unit test coverage.

Simple design Partial Upfront, complex design is hard to resist, especially withoutstrong test-driven development and refactoring.

Pair programming Partial Practiced, but not exclusively.

On-site customer Partial No outside customer is providing a business perspective,currently self serving as customer for research products at hand.

Planning game None Have yet to invoke project management side of XP.

a team software development project.

During the transition of legacy code from For-

tran 77 to Fortran 95, a rough guess at a codingstandard was created and used by the entire team.Based on this experience, a more detailed revisionwas created. (See Appendix A on page 16.) Oneduty of the full-time contractor assigned to the teamis to enforce the coding standard as new content iscommitted to the repository. This function is slatedto be replaced by an automated agent that parsesthe source code.

Given a thoughtfully crafted coding standard, im-proved source code readability is a natural benefitthrough consistent indentation, alignment, naming,and commenting conventions. However, the cod-ing standard must be appropriately tailored to theprogramming language. For example, Fortran 95permits declaring an array variable and later di-mensioning it through a separate statement. Thismulti-line variable declaration can be hard to followand can create confusion, thus prompting a line inthe coding standard to place all attributes of thedeclaration on a single line, if possible. Another ex-ample is that the variable names of arguments in thecalling and called routines do not have to match.However, retaining the same names for both im-proves global comprehension of the code and makescode-generated documentation more coherent.

A coding standard also serves as a sentinel againstthe use of vendor-specific language extensions or de-preciated elements of the language that do not lendthemselves to portability across various platforms.For example, Fortran 95 does not contain a com-plete set of intrinsic functions for accessing system-level utilities or timing, but many compiler vendors

offer extensions like system() and etime(), whichare tempting but create portability headaches.

Collective Ownership Anyone canchange anycodeanywhere inthe system atany time.

The ideal situation for team software developmentoccurs when a pair of developers looks at a givenpiece of code and does not feel the need to changethe indentation, and so forth, and furthermore can-not recall whether they wrote the code in the firstplace. No single developer claims code ownership,yet all share responsibility; all source code is eligiblefor changes by any team member. Using a codingstandard is absolutely essential to reach this goal.

Collective code ownership was a completely for-eign concept to team members prior to this project.Initial acceptance of this philosophy came about be-cause the original developer of the Fun2D/3D codewas no longer at Langley, and the current “codesteward” did not feel comfortable claiming the codeas “his.” Both the software development practicesmentioned above and the tools the team uses foreffective collaboration have cemented the idea of col-lective code ownership to the extent that membersfeel comfortable changing the code without askingpermission of another developer.

Due to the team-oriented nature of the project andthe amount of source code involved, a widely used,source code management system is used, the Con-current Versions System (CVS).m CVS oversees acentral repository of the source code and allows eachteam member to concurrently develop and modifysections as needed. Any changes or additions to alocal working copy can then be committed back tothe repository, whereby they will be available to the

mwww.cvshome.org

7 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

entire team.CVS maintains complete documentation of any

changes made during the course of code develop-ment, and previously modified or deleted code canbe resurrected at any time by any member of theteam. In addition, the system allows team membersto work on platforms located virtually anywhere.The use of a software management tool allows fornearly seamless integration of a number of widelyvarying research projects and eliminates the needfor multiple branches of a code.n

Continuous IntegrationIntegrate andbuild the

system manytimes a day,every time a

task iscompleted.

In a team environment that has many developerswho all contribute to a code base on a daily ba-sis, integrating those changes into a common codebase quickly becomes a major undertaking unlessnew code is integrated and tested as soon as practi-cal, preferably within a few hours.

Continuous integration avoids diverging or frag-mented development efforts, in which developers arenot communicating with each other about what canbe shared or reused. Simply stated, everyone needsto work with the latest version of the code base.Making changes to obsolete code causes integrationheadaches.

Originally, developers manually ran the Hefss

test suite during code modification, but not all devel-opers consistently ran the test suite before checkingtheir code modifications into the repository, so anautomated process was sought. At first the Unix-based cron utility was used to check out a freshversion of the CVS repository, to compile the suiteof codes, and to run regression tests on three differ-ent architectures and compilers every night. How-ever, the Extreme Programming community soonreminded the HEFSS team that “[daily builds] arefor winning-challenged people who can’t integrateevery 5 to 15 minutes and run all the tests at everyintegration,” and they went to a true continuous in-tegration mode of operation on dedicated machines.

The continuous integration process restarts thebuild and test process after each successful set oftests. Test results are automatically logged on aweb server, and failures are e-mailed to all devel-opers listing all CVS commits that were performedsince the last successful build. With this system,errors are detected within a couple hours, and theintegration failure e-mail provides a strong source ofpeer pressure on developers to run a range of testsbefore committing changes.o

nThis CVS controlled LATEX document was jointly com-posed by the team using such an approach.

oSee

Small Releases Put a simplesystem intoproductionquickly, thenrelease newversions on avery shortcycle.

Feedback is the core idea behind the small re-leases practice. Get the software out there and learnfrom it. Strive to make the transition from puresoftware development to software maintenance asquickly as possible. Small releases are enabled byother practices like simple design, automated test-ing, and continuous integration.

The source code management system describedpreviously enables the team to automatically cre-ate releases by merely “tagging” snapshots of therepository for which all the tests pass successfullyduring the continuous integration cycle. So rou-tinely, the team is typically making several releasesthroughout any given day. This snapshot featurealso facilitates the management of releases to out-side users by providing accurate technical supporttailored specifically to the exact source code snap-shot released to a given party. Unfortunately, theHefss code currently has some restrictions on itsexternal distribution; however, it is being used inhouse by several people.27

Test-Driven Development Any programfeaturewithout anautomatedtest simplydoesn’t exist.

Since the time to fix a software defect (aka “bug”)scales exponentially with the time lag between in-troduction and detection,28 it is extremely advan-tageous to trap defects as early as possible duringdevelopment.

Previously known as merely “Testing,” this prac-tice has blossomed into a whole field in itself.29 Test-driven development within XP has two components,one centered around developers and the other cen-tered around customers, or end-users. Developerswrite unit tests so that their confidence in the codecan become part of the code itself, while customerswrite acceptance tests so that their confidence in thecode’s capabilities can also become part of the code.These automated tests allow confidence in the codeto grow over time, allowing the code to become morecapable of accepting change.

Unit tests are intended to verify small quanta offunctionality within a code and should be automatedand run to completion in fractions of a second. Theunit tests serve as a development guide by specify-ing the desired capability, interfaces, and expectedoutput of a functional unit. Unit tests also serve asmobility enablers during code architecture shifts toensure a safe path was taken. Mobility allows code tobe easily reused and to have functionality extendedwhile safely maintaining current functionality. Notethat in most cases, there will be more lines of unit

www.martinfowler.com/articles/continuousIntegration.htmlfor more information.

8 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

test code than actual production code.Acceptance tests check the interactions between

code elements that unit tests cannot cover and doc-ument the existence of a particular code feature.Preferably, customers write acceptance tests.

Since the Hefss code contains active research inmany different disciplines that coexist in the sameframework, work in one field can introduce errorsin others through the common framework. Theseerrors can go unnoticed if the code, in part and inwhole, is not verified in a repeatable manner. Onewell-known approach to finding defects and ensuringthat the code produces repeatable, verified answersis through automated testing.

For example, an unforeseen interaction with mod-ule A is introduced by modifying code in module B.If the problem in module A goes undetected for amonth, it may be difficult to link the problem to aninteraction with module B or to other code modifi-cations made during that month. If the problem inmodule A is detected in minutes by an automatedtesting framework, the interaction of module A andmodule B can be clearly identified before other codemodifications cloud the picture.

The current project began with legacy codethat did not contain a single unit test. Becauseretrofitting an exhaustive set of unit tests to the ex-isting legacy code was deemed too expensive, theoriginal intent was to introduce unit tests as newcode was added and old code was refactored. Todate, however, unit testing has not been widelyadopted by the team despite the creation of a unittesting framework for Fortran 95.p Currently, unittests only cover a very small percentage of the codebase. However, significant unit testing coverage isbeing built into Ruby-based wrappers used for test-ing and grid adaptation. Additionally, some of thelow-level Fortran library routines are becomingtest-infected, for example, character-to-number con-version routines and linear algebra routines.

The acceptance tests for the Hefss code are asuite of over 240 regression tests performed by aseries of Makefiles. These regression tests simplycompare the convergence history of residual, force,and moment calculations (or other output appropri-ate to the code under test) to previously recordedexecutions to machine precision (not just 2–3 dig-its). These results are referred to as “golden files.”These test fixtures ensure that the current code gives

pTo facilitate both the writing and running of unit tests forFortran 95 source code, a testing framework called F95unit

has been developed using Ruby. F95unit has a model sim-ilar to the unit-testing frameworks for other languages, e.g.,JUnit, PyUnit, Ruby test/unit.

the same discrete answer as the original golden file.Makefiles were initially selected to perform thesetests because the tests were seen as a natural exten-sion to code compilation.q The compile operationswere incorporated into the tests, so the tests arealways performed with an executable file producedfrom the current source files. Test cases can be runon an individual basis or as an entire suite.

The current set of acceptance tests for Hefss wasadded incrementally to first cover the legacy func-tionality of Fun3D and then new functionality, as itwas added to the suite. The Makefiles that performthe tests have become complex, hard to maintain,and are being replaced in an incremental fashionwith unit-tested Ruby. This unit-tested Ruby frame-work should be much easier to maintain and allowmore flexibility. The Ruby framework can be reusedto link a number of the codes together to performcomplex functions such as design optimization andgrid adaptation, in addition to testing.

Refactoring Programmersrestructurethe systemwithoutchanging itsbehavior toremoveduplication,improve com-munication,simplify, oraddflexibility.

To extend a code’s viable lifetime and strive forthe simplest design that will work, developers needlots of practice modifying the design, so that whenthe time comes to change the system, they will notbe afraid to try it. Constant refactoring is abso-lutely essential to keeping the cost-of-change curvefrom growing exponentially as time increases. Refer-ence 30 teaches developers how to refactor and why.

Automated testing, as discussed earlier, is abso-lutely essential to refactoring. Without a safetynet of tests, subtle shifts in the code’s fundamentalarchitecture toward a more agile, clean, and under-standable design is extremely difficult and frustrat-ing. Testing allows developers to modify code thatthey did not write so that the original developer canbe sure that modified routines still perform the orig-inal purpose correctly, if the appropriate unit testspass. This process leads to an environment in whichthe tests are paramount and the code can be easilymodified to add new functionality, improve speed,or become more readable.

Due in part to the lack of extensive unit testing inthe Hefss code, many refactorings are delayed, cre-ating a backlog of work. Occasionally, the team willtackle some of these tasks, but so far the backlogcontinues to grow. A renewed effort at promot-ing the benefits of test-first programming is beingmade within the team by drawing attention to theinefficiencies inherent in the “Code-n-Fix” style ofprogramming.

qIf the code is modified and needs to be recompiled, itshould also be tested.

9 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Simple DesignThe systemshould be

designed assimply as

possible atany givenmoment;

extracomplexity is

removed assoon as it isdiscovered.

Simple design is defined by two ideas: One is theYAGNI principal, otherwise known as, “you aren’tgonna need it,” and the other is a chant, “do thesimplest thing that could possibly work.” Theseprincipals should be internalized and provide instinc-tive reactions to “gold plating” or other ideas thatdo not seem to fit the current task. A simple designshould not contain ideas that are not used yet butthat are expected to be used in the future. However,one should pay attention to the word “expected.” Ifyou are somehow assured of the future, and that agiven idea will be necessary, design with it in mind,but do not implement it now because you will bestknow how to add it when the time comes.

As with refactoring, the lack of unit test coveragewithin Hefss code makes this practice difficult tofollow completely. For many developers, it is alsotypically contrary to years of prior practice; regard-less, the team can now at least recognize complexityand several major strides have been made to reduceexisting manifestations.

Pair ProgrammingAllproduction

code iswritten with

twoprogrammers

at onemachine.

The initial reaction to the idea of two peopleworking on the same task at the same computer atthe same time is usually negative. However, thisreaction is typically caused by painful experiencesassociated with “pair debugging” or simply misun-derstanding the true nature of pair programmingitself. Pair programming is not one person pro-gramming while another person watches. It is moreakin to an animated conversation, facilitated by awhite board, where one participant might grab themarker from the other and make a change whilethe first is still talking. Pair programming shouldbe highly dynamic, and the participants should beable to switch “driver” and “navigator” roles at anypoint. Besides making programming more fun, pairprogramming provides an extensive host of bene-fits, such as streamlining communication, propagat-ing knowledge, and continuous code reviews. Pairprogramming also greatly enhances collective codeownership. For a detailed discussion of the art ofpair programming, see Reference 31.

Within the HEFSS team, frequent pair program-ming is highly encouraged but not mandated. Itis used for all aspects of code development, for ex-ample, debugging, teaching, refactoring, and addingnew features. Intimately involving a number of re-searchers at the lowest levels of code developmentensures a relatively high truck number.r Traditional

rThe truck number is the size of the smallest setof people in a project such that, if all of them got

CFD codes at Langley are developed by individualsor small teams and most of the resulting code basehas a truck number of 1 or perhaps 2, whereas thecurrent collaborative team approach yields a valuenear 10.

On-Site Customer Include areal, live useron the team,availablefull-time toanswerquestions.

This XP practice is intended to remove the com-munication barriers present in a typical contractedpiece of software where a slew of requirements andspecifications are defined upfront and then the “codemonkeys” are let loose to grind out the requiredpiece of software. The pitfalls with this sort of con-tract negotiation are many, the least of which is thatthe customers seldom know what they want beforethey see a working prototype. By placing an enduser with the team, XP is nearly guaranteed of de-livering a relevant, useful piece of software.

As discussed in Reference 11, the scientific re-search environment often creates a situation in whichthe developers are their own customers. This sce-nario requires diligent role playing to keep technicaland business needs separated. Currently, the HEFSSteam members largely act as their own customers,with only very minor input from project stakehold-ers.

The Planning Game Quicklydeterminethe scope ofthe nextrelease bycombiningbusinesspriorities andtechnicalestimates; asrealityovertakes theplan, updatethe plan.

XP uses a four-dimensional space to plan andmeasure progress: time, cost, quality, and scope.Scope is typically ignored by many project planningmechanisms, but it plays a central role in XP. Theplanning game has two levels: iteration planning andrelease planning. The basic premise of the planninggame is that business people determine scope, pri-ority, composition of releases, and dates of releases,while technical people provide estimates, design con-sequences, the process, and detailed scheduling.

As shown in Table 2 on page 7, the HEFSS teamhas not yet begun using this practice. However, full-cost accounting practices now being put into placemay force this final XP practice to be invoked.

Project Retrospectives

Sometimes referred to as XP’s “thirteenth prac-tice,” project retrospectives32 are important com-ponents of tailoring a process to a given situation.Every few months, the team takes time to reflect onpast events and accomplishments. The goal is notfaultfinding, but instead the goal is to learn how todo better in the future. During these sessions, theteam begins with a discussion guided by the follow-ing three questions: what has gone well? what could

hit by a truck, the project would be in trouble. Seec2.com/cgi/wiki?TruckNumber for further discussion.

10 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

be improved? and with what new techniques or toolsshould the team investigate? Currently these ses-sions are not as formal or wide-reaching as some ofthe formats presented in Reference 32.

Scrum Status Meetings

A daily, stand-up meeting is normally associatedwith XP, but it is not explicitly called out as a prac-tice or given much structure, except that nobodycan sit during the meeting, it should be short, andit should happen every day before developers startpair programming. The HEFSS team has adopted asimilar, but more structured status meeting formatfrom another agile methodology, Scrum.8

A Scrum status meeting is held daily by an ap-pointed “Scrum Master” and lasts no longer than15 minutes. The meeting has an open attendancepolicy, but only team members are allowed to talk.The team members, in turn, succinctly report threethings: what they did since the last meeting, whatthey will do by the next meeting, and what got inthe way (impediments). Additional discussion dur-ing a Scrum is strictly limited to clarification-relatedquestions and to note topics that will be discussedat a later time by interested parties. The Scrummaster plays the role of gatekeeper and takes notes.Later, the Scrum master compares performance withpast commitments and follows up on situations thatappear to be stalled. Most importantly, the Scrummaster is responsible for removing impediments.

Scrum status meetings have several benefits froma management perspective. They offer a quick andeasy mechanism to collect data for status reportsand yield an immediate sense of whether a team isin trouble. By using Scrums to their benefit, man-agement can avoid what Peopleware33 claims is theultimate management sin: wasting people’s time.

Since the HEFSS team is currently dispersedthroughout the local campus and most developersare not full time, the Scrum status meeting is onlyheld weekly. In addition, the team also allots sometime afterward to address any topics which may havearisen during the Scrum. This post-Scrum gather-ing is governed by Open Space’s Law of Two Feet,s

which states that if during the course of any gather-ing, persons find themselves in a situation in whichthey are neither learning nor contributing, they mustuse their two feet and go to some more productivespace.

sSee www.openspaceworld.com/users guide.htm for morediscussion.

Other Communication Mechanisms

Since communication and cooperation are essen-tial to the success of the effort, several additionaltools are employed in addition to the communi-cation mechanisms implicit in XP. The first isa Majordomo-based electronic mailing list which,serves to facilitate communication among teammembers that are distributed across the local cam-pus. In addition to the E-mail list and weekly meet-ings, the team also uses a web-based collaborativetool known as a Wiki.t A Wiki allows users to freelycreate and edit web page content using any webbrowser. Wikis have a simple text syntax for creat-ing web page elements, and they dynamically createa new web page when they encounter a CamelCasedword (a mixed-case word containing at least two cap-itals). The team uses the Wiki for a number ofpurposes. For example, the testing status page iscontained in the Wiki so that adding new data tothe page can be done by anyone. The Wiki is alsoused to share data for emerging test cases that haveyet to be incorporated into the automated testingsystem, and it also serves as a repository for other-wise tacit knowledge, for example, CompilerNotes,AvoidingSshPasswords, CreatingNewTestCases.

Documentation

Documentation for the Hefss code takes manyforms. While currently the Hefss code itself lacks aformal users’ manual,u it does have a more exactingform of documentation, a large set of regression testcases. Each test case directory contains everythingneeded to run a given type of case and can usuallybe readily adapted to a new type of case.

Meanwhile, developers have three tools availablefor browsing the Hefss code base. Code browsingcan take many forms and be done for various rea-sons; consolidating them into a single tool has so farproven to be an elusive goal.

The simplest tool is a web-based rendering of theCVS repository, generated on-the-fly by the opensource ViewCVS

v tool. This approach is basedon the CVS repository’s file directory structure andthus lacks the ability to navigate the source by us-ing internal structure. However, it is the only toolthat readily provides access to prior versions of thesource code.

A second tool, developed by a support servicecontractor using C++, parses the source code andgenerates web-based output by using a commercial

twww.wiki.orguThe users’ manual is being written.vviewcvs.sourceforge.net

11 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

tool, Understand for Fortran,w which extractscalling tree graphs and code statistics. The C++code also creates tables of variable declarations andrenders comments associated with routines that areplaced according to the coding standard. The webpages generated by this tool include source code list-ings that have been formatted with line numbers andare keyword-colored to enhance readability.

A third code-browsing opportunity leverages theopen source code documentation system, RDoc,x

which was originally intended for documenting Rubysource. A short extension for this system was writ-ten to parse and format Fortran 95y and has sub-sequently been accepted into the RDoc distribution.The RDoc system extracts a graph of the code sourcebased on files, modules, and routines. From thesedata, it can generate frame-based web pages, XML,or Windows help files that can be used to navigatethe calling structure.

Research ProductsTo illustrate some specific examples of lessons

learned during the current effort, several researchfocuses are discussed briefly below. The team’s ex-perience has been largely a positive one; however, itis clear that properties of modularity and extensibil-ity are earned through experience and not designedinto a system upfront.

Time-Accurate Simulations

In support of both passive and active flow con-trol research at Langley, the perfect-gas capabilitiesin the solver have been extended to higher ordertemporal accuracy. The validity of the approachhas been verified through numerical experiments inwhich an order property consistent with a second-order scheme has been demonstrated for turbulentflows. With a trivial amount of effort, the mod-ifications required to obtain these results in theperfect-gas realm were extended to include reacting-gas simulations. Current work is focused on evaluat-ing third- and fourth-order time-integration schemesfor perfect-gas flows,34 which should also be readilyextendable to more complicated physical models asneeded.

Incorporating Multiple Element Types

Initially, the Hefss solver made sole use of tetra-hedral element types to discretize a given domain.However, the ability to accommodate additional ele-ment types such as prisms, hexahedra, and pyramids

wwww.scitools.com/uf.htmlxrdoc.sourceforge.netyThis extension was accomplished with only 120 lines of

code.

provides greater flexibility to match a given elementtype to a particular flow topology, and the extensionto include such elements in all aspects of the pack-age is currently ongoing. This effort represents oneof, if not the, most substantial modifications to thesoftware to date since it extends the fundamentaldata structure used throughout the code base. Thepre-/post-processor and solvers, as well as all of theirassociated linearizations for optimization and adap-tation, require considerable modification at the mostfundamental levels. This undertaking has revealedmany areas in which additional refactoring is stillrequired before an acceptable level of modularity isachieved.

Two-Dimensional Capability

A major advantage of pursuing mixed-elementdiscretizations is the ability to recover a trulytwo-dimensional solution capability, which can beachieved through the use of prismatic and hexahe-dral elements in the spanwise direction, such thatflux balances need only be performed in the plane ofsymmetry. Axisymmetric flows can also be readilyaccommodated by adding source terms. The ben-efits of such an approach are substantial, in that aseparate code need not be maintained for such prob-lems, a longtime burden for the original Fun2D/3Ddevelopers. In addition, all algorithms and physi-cal models available in the three-dimensional pathare immediately available for two-dimensional so-lutions, which allows basic research to be carriedout on less costly two-dimensional problems. Whencomputations are extended to three dimensions, theinconsistencies normally associated with switchingbetween two separate solvers are no longer an issue,and the results are not contaminated by differencesin discretizations or solution methods.

Multigrid Algorithms

A major thrust of the FAAST project is aimed atachieving textbook multigrid efficiency (TME), aneffort that could drastically reduce solution timesfor complex problems.35 Since the baseline unstruc-tured solver used as the foundation for the currentwork did not include options for multigrid accelera-tion, much work has focused on implementing sucha capability.

The use of an agglomeration multigrid algorithmrelies on an edge-based discretization of the govern-ing equations; this requirement precludes the abil-ity to compute solutions to the full Navier-Stokesequations on mixed-element grids. For this rea-son, a geometric non-nested multigrid approach hasbeen initially chosen for the Hefss solver. Opera-tions such as coarse-grid partitioning and intergrid

12 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

transfers in a complex domain-decomposed environ-ment have been developed, and a simple FMG/FASmultigrid algorithm has been implemented. Al-though this capability has been coded primarilywith perfect-gas applications in mind, the schemehas been implemented such that users perform-ing reacting-gas computations will also be able tomake immediate use of this research without theneed to duplicate extensive low-level code develop-ment typically associated with geometric multigridon domain-decomposed unstructured meshes.

One component necessary to achieve TME is aline-implicit solver to overcome stiffness associatedwith high-aspect ratio grid elements. The ability toform lines suitable for implicit relaxation, to obtainan appropriate partitioning, and to perform an ex-act inversion along each line has been developed andis applicable to any set of physical equations beingsolved.15

Incorporating High-Energy Physics

The thermochemical nonequilibrium models inHefss are identical to those in Laura, but theirimplementation is substantially different. Laura

made extensive use of precompiler directives thatallocated memory and defined the code path accord-ing to a diverse set of options. This compilationstrategy evolved from an absence of dynamic mem-ory allocation capability in Fortran when Laura

was originally coded and because of a desire to com-pletely eliminate any model-dependent conditionalstatements within loops that could compromise vec-tor efficiency. Any change in the gas model requireda recompilation of the source code. Laura employsa script to guide a user through the various permuta-tions and combinations of options, but the process isburdensome to a user conducting parametric studies.In contrast, Hefss only needs to be compiled onceon any platform, regardless of the desired physicsmodel options.

Model parameters in Laura are initialized inblock data routines; these routines have been re-placed by formatted data files that use conventionalformatted reads and namelists in the Hefss solver.Model parameters that are unlikely to be changed bythe user (thermodynamic curve fit constants, speciesmolecular weights, and heats of formation) are as-sembled in one set of data files. Gas model optionsthat are likely to be changed by the user on a fre-quent basis, such as the chemical composition of thegases entering the domain or the thermochemicalmodel, are assembled in a separate file. This sep-aration minimizes the amount of setup required toperform a given analysis.

Adjoint Solver and Sensitivity Analysis

As important as the software practices in thiseffort are to the development of new analysis ca-pabilities, they are absolutely critical to the suc-cess of the design element under FAAST. In Ref-erences 13, 14, 15, a discrete adjoint capability hasbeen developed for the solver. This effort representsthe only capability of its kind and relies on severalhundred thousand lines of exact hand-differentiatedlinearizations of the preprocessor, flow solver, andmesh movement codes with respect to both the de-pendent variables and the grid coordinates. Forfree-stream conditions of Mach 0.84, a 3.06 degreeangle of attack, and a Reynolds number of 5 M, sen-sitivity derivatives of the lift and drag coefficients,with respect to several shape design variables forfully turbulent flow over an Onera M6 wing,36 thatwere computed by using the discrete adjoint formu-lation, are shown in Table 3 on the next page. Theadjoint results are in excellent agreement with thoseobtained using a complex-variable approach37 witha step size of 1×10−30. This accuracy can easilybe compromised by a single error anywhere in thesource code. With a dozen researchers modifyingcode on a daily basis, the use of continuous integra-tion and automated testing is critical in maintainingsuch accuracy. Just as residual and force conver-gence histories are monitored to machine accuracyfor the flow solver on several architectures, simi-lar quantities are constantly tested for the adjointsolver and gradient evaluation codes. This constanttesting ensures that discrete consistency between theanalysis and design tools is always maintained, re-gardless of the modifications being implemented inother parts of the software.

Similar to the continuous integration and testingperformed for the hand-differentiated code, a Rubycode has been developed similar to the effort de-scribed in Reference 38 to automatically convert thecodes in the Hefss suite to a complex-variable for-mulation. This capability can immediately recover aforward mode of differentiation for the entire solverat any time, with no user intervention. This proce-dure is also continuously tested.

Design Optimization

Approximation and Model Management Opti-mization (AMMO) techniques39,40,41 have been re-cently added to the Hefss software set. AMMOis a methodology aimed at maximizing the use oflow-fidelity models in iterative procedures with oc-casional but systematic recourse to higher-fidelitymodels for monitoring the progress of the algorithm.In current demonstrations, AMMO has exhibited

13 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Table 3 Comparison of discrete adjoint and complex variable design variable derivatives for coefficients of liftand drag for fully turbulent flow over an Onera M6 wing.

Camber Thickness Twist Shear

CL0.956208938269467 -0.384940321071468 -0.010625997076936 -0.005505627646872 discrete0.956208938269046 -0.384940321071742 -0.010625997076937 -0.005505627647001 complex

CD0.027595818243822 0.035539494383655 -0.000939653505699 -0.000389373578383 discrete0.027595818243811 0.035539494383619 -0.000939653505699 -0.000389373578412 complex

from three to five-fold savings in terms of high-fidelity simulations on aerodynamic optimization of3D wings and multi-element airfoils, where simpli-fied physics models (e.g., Euler) computed on coarsegrids serve as low-fidelity models, while more accu-rate models (e.g., Navier-Stokes) computed on finergrids serve as high-fidelity models. AMMO wasthe first approach for using variable-fidelity modelsanalytically guaranteed to converge to high-fidelityanswers.

Because AMMO relies on using a variety of modelsin a single optimization run, maintaining continuousintegration and consistency with the entire softwareset is especially crucial for obtaining stable optimiza-tion results. However, designing a testing strategyfor optimization presents an interesting challengesince optimization algorithms requires reasonablywell converged analyses and is, therefore, expensive.Procedures for automated testing of optimizationsoftware is currently under development.

Output Error Correction and Grid Adaptation

One of the thrusts of the FAAST program is todevelop a mathematically rigorous methodology toadapt a grid discretization to directly improve thecalculation of an output function. An adjoint-basederror correction and adaptation scheme has pro-duced excellent results in 2D.19 This scheme is beingextended to 3D and incorporated into Hefss.21 Thiserror correction and adaptation scheme requires thecalculation of flow and adjoint residuals on embed-ded grids with interpolated solutions. The modu-larity of the Hefss reconstruction, flux, and adjointroutines facilitated this calculation.

The interpolation of the solution onto the em-bedded grid requires the calculation of least-squaresgradients. This gradient routine was readily sharedbetween the flow and adjoint codes. The element-based interpolation scheme was developed test-firstwith the F95unit framework. The code to computethe flow and adjoint residuals consists of only a smalldriver routine; the remainder of the code is reusedfrom the flow and adjoint solvers. The anisotropicadaptation metric is calculated with code that wasalso developed test-first using the F95unit frame-work.

Concluding RemarksWhile the FAAST team has learned a great deal

from the arena of commercial software development,it is important to remember that Langley’s primarygoal is to advance the state of the art rather thandeliver commercial software products. However, ithas been the experience of the team that the syn-thesis and extension of the Center’s computationalfluid dynamic capabilities has greatly enhanced theability to perform research. Development of a uni-fied framework for computational simulation enablesresearchers to examine a number of widely vary-ing research disciplines and to apply new technologyin a more straightforward and encompassing fash-ion. The capability laid out in this effort currentlysupports a broad range of research projects and ap-plications, including general unstructured-grid algo-rithms, high-energy flows, mixed-element computa-tions, error estimation, grid adaptation, design op-timization, time-accurate schemes, turbulence mod-eling, and multigrid algorithms.

As advances are made in each area, they immedi-ately become part of the mainstream capability andare readily available to other researchers and users.Some team members have expressed concern aboutusing such a dynamic software tool for work on spe-cific research projects. However, these concerns aremitigated by developing and continuously invokinga suite of automated test cases.

One truly remarkable aspect of this project is thata gaggle of developers which typically shuddered atany mention of the word “process” gelled into a teamof developers using a fairly rigorous, pervasive soft-ware process that they enjoy.

AcknowledgmentsThe authors would like to recognize the rest of

the FAAST team that made contributions to thispaper: Natalia Alexandrov, Harold Atkins, KarenBibb, Robert Biedron, Mark Carpenter, Dana Ham-mond, William Jones, Elizabeth Lee-Rausch, TomRoberts, James Thomas, Veer Vatsa, Sally Viken,and Jeffery White.

The authors would also like to thank CharlesMiller of the Aerothermodynamics Branch at NASA

14 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Langley Research Center, Hampton, Virginia, forenduring multiple manuscript reviews of this work.

ColophonThis paper was typeset in Donald Knuth’s 10pt

Computer Modern Font using the free, multi-platform LATEX typesetting system and Kleb’s aiaabundle.42 The following packages were also used:array, dcolumn, tabularx, multirow, xspace, varioref,fancyvrb, url, and textcomp.

References1Thomas, J. L., Alexandrov, N., Alter, S. J., Atkins,

H. L., Bey, K. S., Bibb, K. L., Biedron, R. T., Carpenter,M. H., Cheatwood, F. M., Drummond, P. J., Gnoffo, P. A.,Jones, W. T., Kleb, W. L., Lee-Rausch, E. M., Merski, N. R.,Mineck, R. E., Nielsen, E. J., Park, M. A., Pirzadeh, S. Z.,Roberts, T. W., Samareh, J. A., Swanson, R. C., Vatsa, V. N.,Weilmuenster, K. J., White, J. A., Wood, W. A., and Yip,L. P., “Opportunities for Breakthroughs in Large-Scale Com-putational Simulation and Design,” NASA/TM 2002-211747,June 2002.

2Cheatwood, F. M. and Gnoffo, P., User’s Manual for theLangley Aerothermodynamic Upwind Relaxation Algorithm(LAURA), NASA TM-4674, 1996.

3White, J. A. and Morrison, J. H., “A Pseudo-TemporalMulti-Grid Relaxation Scheme for Solving the ParabolizedNavier-Stokes Equations,” AIAA Paper 99-3360, June 1999.

4Anderson, W. K. and Bonhaus, D. L., “An ImplicitUpwind Algorithm for Computing Turbulent Flow on Un-structured Grids,” Computers & Fluids, Vol. 23, No. 1, Jan.1994, pp. 1–21.

5Anderson, W. K., Rausch, R. D., and Bonhaus, D. L.,“Implicit/Multigrid Algorithms for Incompressible TurbulentFlows on Unstructured Grids,” Journal of ComputationalPhysics, Vol. 128, No. 2, 1996, pp. 391–408.

6Frink, N., “Tetrahedral Unstructured Navier-StokesMethod for Turbulent Flow,” AIAA Journal , Vol. 36, No. 11,Nov. 1998, pp. 1975–1982.

7Cambier, L. and Gazaix, M., “elsA: An Efficient Object-Oriented Solution to CFD Complexity,” AIAA Paper 2002-0108, Jan. 2002.

8Schwaber, K. and Beedle, M., Agile Software Develop-ment with Scrum, Prentice Hall, Oct. 2001, See also http:

//www.controlchaos.com last accessed 16 June 2003.9McBreen, P., Software Craftsmanship: The New Imper-

ative, Addison-Wesley, 2002.10Wood, W. A. and Kleb, W. L., “Extreme Programming

in a Research Environment,” Extreme Programming and Ag-ile Methods—XP/Agile Universe 2002 , edited by D. Wellsand L. Williams, Vol. 2418 of Lecture Notes in ComputerScience, Springer-Verlag, Chicago, IL, Aug. 2002, pp. 89–99.

11Wood, W. A. and Kleb, W. L., “Exploring XP for Sci-entific Research,” IEEE Software, Vol. 20, No. 3, May 2003,pp. 30–36.

12Bibb, K. L., Peraire, J., and Riley, C. J., “HypersonicFlow Computations on Unstructured Meshes,” AIAA Paper97-0625, Jan. 1997.

13Nielsen, E. J., Aerodynamic Design Sensitivities on anUnstructured Mesh Using the Navier-Stokes Equations anda Discrete Adjoint Formulation, Ph.D. thesis, Virginia Poly-technic Institute and State University, 1998.

14Nielsen, E. J. and Anderson, W. K., “Recent Improve-ments in Aerodynamic Design Optimization on UnstructuredMeshes,” AIAA Journal , Vol. 40, No. 6, June 2002, pp. 1–9,See also AIAA Paper 01-0596.

15Nielsen, E. J., Lu, J., Park, M. A., and Darmofal,D. L., “An Exact Dual Adjoint Solution Method for Turbu-lent Flows on Unstructured Grids,” AIAA Paper 2003-0272,Jan. 2003.

16Venditti, D. A. and Darmofal, D. L., “Adjoint ErrorEstimation and Grid Adaptation for Functional Outputs: Ap-plication to Quasi-One-Dimensional Flow,” Journal of Com-putational Physics, Vol. 164, 2000, pp. 204–227, See alsoAIAA Paper 99-3292.

17Venditti, D. A. and Darmofal, D. L., “Grid Adaptationfor Functional Outputs: Application to Two-Dimensional In-viscid Flows,” Journal of Computational Physics, Vol. 176,2002, pp. 40–69, See also AIAA Paper 2000-2244.

18Venditti, D. A., Grid Adaptation for Functional Out-puts of Compressible Flow Simulations, Ph.D. thesis, Mas-sachusetts Institute of Technology, 2002.

19Venditti, D. A. and Darmofal, D. L., “AnisotropicGrid Adaptation for Functional Outputs: Application toTwo-Dimensional Viscous Flows,” Journal of ComputationalPhysics, Vol. 187, 2003, pp. 22–46.

20Park, M. A., “Adjoint-Based, Three-Dimensional ErrorPrediction and Grid Adaptation,” AIAA Paper 2002-3286,June 2002.

21Park, M. A., “Three–Dimensional Turbulent RANSAdjoint–Based Error Correction,” AIAA Paper 2003-3849,June 2003.

22Matsumoto, Y., Ruby in a Nutshell , O’Reilly, Se-bastopol, CA, 2002.

23Thomas, D. and Hunt, A., Programming Ruby: ThePragmatic Programmer’s Guide, Addison-Wesley, 2001.

24Beck, K., Extreme Programming Explained , Addison-Wesley, 2000.

25Paulk, M. C., Weber, C. V., and Curtis, W., The Capa-bility Maturity Model: Guidelines for Improving the SoftwareProcess, Addison-Wesley, 1995.

26Humphrey, W. S. and Lovelace, M., Introduction to theTeam Software Process, Addison-Wesley, 1999.

27Lee-Rausch, E. M., Buning, P. G., Morrison, J. H., Park,M. A., Rivers, S. M., Rumsey, C. L., and Mavriplis, D.,“CFD Sensitivity Analysis of a Drag Prediction WorkshopWing/Body Transport Configuration,” AIAA Paper 2003-3400, June 2003.

28Boehm, B. W., Software Engineering Economics, Pren-tice Hall, 1st ed., Oct. 1981.

29Beck, K., Test Driven Development: By Example,Addison-Wesley, 2002.

30Fowler, M., Refactoring: Improving the Design of Exist-ing Code, Addison-Wesley, 1999.

31Williams, L. and Kessler, R., Pair Programming Illumi-nated , Addison-Wesley, 2003.

32Kerth, N., Project Retrospectives: A Handbook for TeamReviews, Dorset House, 2002.

33DeMarco, T. and Lister, T., Peopleware: ProductiveProjects and Teams, Dorset House, 2nd ed., 1999.

34Carpenter, M. H., Viken, S. A., and Nielsen, E. J., “TheEfficiency of High Order Temporal Schemes,” AIAA Paper2003-0086, Jan. 2003.

35Thomas, J. L., Diskin, B., and Brandt, A., “TextbookMultigrid Efficiency for Fluid Simulations,” Annual Reviewof Fluid Mechanics, Vol. 35, 2003, pp. 317–340.

15 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

36Schmitt, V. and Charpin, F., “Pressure Distributionson the ONERA-M6 Wing at Transonic Mach Numbers,”AGARD AR-138, May 1979.

37Anderson, W. K., Newman, J. C., Whitfield, D. L., andNielsen, E. J., “Sensitivity Analysis for the Navier-StokesEquations on Unstructured Meshes Using Complex Vari-ables,” AIAA Journal , Vol. 39, No. 1, Jan. 2001, pp. 56–63,See also AIAA Paper 99-3294.

38Martins, J. R. R. A., Kroo, I. M., and Alonso, J. J., “AnAutomated Method for Sensitivity Analysis Using ComplexVariables,” AIAA Paper 2000-0689, Jan. 2000.

39Alexandrov, N. M. and Lewis, R. M., “A Trust RegionFramework for Managing Approximation Models in Engineer-ing Optimization,” Tech. Rep. AIAA Papers 96-4101 and96-4102, Sept. 1996.

40Alexandrov, N. M., Nielsen, E. J., Lewis, R. M., and An-derson, W. K., “First-order model management with variable-fidelity physics applied to multi-element airfoil optimization,”AIAA Paper 2000-4886, Sept. 2000.

41Alexandrov, N. M., Lewis, R. M., Gumbert, C. R.,Green, L. L., and Newman, P. A., “Approximation and ModelManagement in Aerodynamic Optimization with Variable-Fidelity Models,” Journal of Aircraft , Vol. 38, No. 6,November–December 2001, pp. 1093–1101.

42Kleb, W. L., “aiaa—A LATEX Class and BibTEXStyle for AIAA Conference Papers and Journal Submis-sion/Simulation,” Electronic Documentation, April 1999,Version 2.5.

43Alexandrov, N., Atkins, H. L., Bibb, K. L., Biedron,R. T., Gnoffo, P. A., Hammond, D. P., Jones, W. T., Kleb,W. L., Lee-Rausch, E. M., , Nielsen, E. J., Park, M. A., Ra-man, V. V., Roberts, T. W., Thomas, J. L., Vatsa, V. N.,Viken, S. A., White, J. A., and Wood, W. A., “Team Soft-ware Development for Aerothermodynamic and AerodynamicAnalysis and Design,” NASA/TM 2003-212421.

Appendix A

Coding Standard

Note: parenthetical numbers refer to line numbers in thesample program which follows.

Style

� Free format with no character past column 80

� Indentation: begin in first column and recursively indentall subsequent blocks by two spaces.

� Start all comments within body of code in first column[42].

� Use all lowercase characters; however, mixed-case maybe used in comments and strings.

� Align continuation ampersands within code blocks [77].

� No tab characters

� Name ends [85].

Comments

� For cryptic variable names, state description using by acomment line immediately preceding declaration or onend of the declaration line [62].

� For subroutines, functions, and modules, insert a con-tiguous comment block immediately preceding declara-tion containing a brief overview followed by an optionaldetailed description [42].

Variable Declarations

� Do not use Fortran intrinsic function names.

� Avoid multi-line variable declarations.

� Declare intent on all dummy arguments [63].

� Declare the kind for all reals, including literal constants,using a kind definition module.

� Declare dimension attribute for all non-scalars [63].

� Line up attributes within variable declaration blocks.

� Any scalars used to define extent must be declared priorto use [60].

� Declare a variable name only once in a scope, includinguse module statements.

Module Headers

� Declare implicit none [35].

� Include a public character parameter containing theCVS $Id$ tag [37].

� Include a private statement and explicitly declare pub-lic attributes.

Subroutines and Functions

� The first executable line should be continue [69].

� Use the only attribute on all use statements [58].

� Keep use statements local, i.e., not in the moduleheader.

� Group all dummy argument declarations first, followedby local variable declarations.

� All subroutines and functions must be contained withina module.

� Any pointer passed to a subroutine or function must beallocated by at least size 1 to avoid null or undefinedpointers.

Control Constructs

� Name control constructs (e.g., do, if, case) which spana significant number of lines or form nested code blocks.

� No numbered do-loops.

� Name loops that contain cycle or exit statements.

� Use cycle or exit rather than goto.

� Use case statements with case defaults rather than if-constructs wherever possible.

� Use F90-style relational symbols, e.g., >= rather than.ge. [73].

Miscellaneous

� In the interest of efficient execution, consider avoiding:

– assumed-shape arrays

– derived types in low-level computationally inten-sive numerics

– use modules for large segments of data

� Remove unused variables.

� Do not use common blocks or includes.

16 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Illustrative Example

1 ! Define kinds to use for reals in one place

23 module kind_defs

45 implicit none

67 character (len=*), parameter :: kind_defs_cvs_id = &

8 ’$Id: cs_example.f90,v 1.5 2002/08/13 02:37:59 kleb Exp $’

910 integer, parameter :: sp=selected_real_kind(P=6) ! single precision

11 integer, parameter :: dp=selected_real_kind(P=15) ! double precision

1213 end module kind_defs

1415 ! A token module for demonstration purposes

1617 module some_other_module

1819 implicit none

2021 character (len=*), parameter :: some_other_module_cvs_id = &

22 ’$Id: cs_example.f90,v 1.5 2002/08/13 02:37:59 kleb Exp $’

2324 integer, parameter :: some_variable = 1

2526 end module some_other_module

2728 ! A collection of transformations which includes

29 ! stretches, rotations, and shearing. This comment

30 ! block will be associated with the module declaration

31 ! immediately following.

3233 module transformations

3435 implicit none

3637 character (len=*), parameter :: transformations_module_cvs_id = &

38 ’$Id: cs_example.f90,v 1.5 2002/08/13 02:37:59 kleb Exp $’

3940 contains

4142 ! Computes a stretching transformation.

43 !

44 ! This stretching is accomplished by moving

45 ! things around and going into a lot of other details

46 ! which would be described here and possibly even

47 ! another "paragraph" following this.

48 !

49 ! This contingous comment block will be associated with the

50 ! subroutine or function declaraion immediately following.

51 ! It is intended to contain an initial section which gives

52 ! a one or two sentence overview followed by one or more

53 ! "paragraphs" which give a more detailed description.

5455 subroutine stretch ( points, x, y, z )

5657 use kind_defs

58 use some_other_module, only: some_variable

5960 integer, intent(in) :: points

6162 ! component to be transformed

63 real(dp), dimension(points), intent(in) :: x, y

64 real(dp), dimension(points), intent(out) :: z ! transformation result

6566 external positive

67 integer :: i

6869 continue

7071 i = 0

7273 if ( x(1) > 0.0_dp ) then

74 call positive ( points, x, y, z )

75 else

76 do i = 1, points

77 z(i) = x(i)*x(i) + 1.5_dp * ( real(i) + x(i) )**i &

78 + ( y(i) * real(i) ) * ( x(i)**i + 2.0_dp ) &

79 + 2.5_dp * real(i) + 148.2_dp * some_variable

80 enddo

81 endif

8283 end subroutine stretch

8485 end module transformations

Appendix B

Fortran 95

The rationale for some elements of the codingstandard presented in the previous section are dis-cussed in this section.

Best Practices

The use of implicit none minimizes the possi-bility of variable type errors. An example of a typeerror is when the implicit Fortran integer typingscheme creates integers for variable names beginningwith the letters “i” through “n” when the user hadintended a real variable. This unintended declara-tion type is avoided because implicit none requiresevery variable to be declared explicitly.

The use of only∗ prevents unintended changes tovalues of other variables in the inherited modules.The only statement also facilitates finding the mod-ule that provides the inherited variable. To furtherrestrict access to variables or subroutines in mod-ules, a private statement is to be placed at thetop of the module. An exclusive and explicit list ofpublic entities is therefore required to share moduledata and methods outside the module. This exclu-sivity prevents unintended variable modifications.

Use of equality comparison with reals should beavoided because small, round-off errors may bepresent. The difference between the two variablesis compared to an intrinsic function like tiny() toprovide a more reliable comparison.

In general, the use of the select case conditionalconstruct is more efficient than using an if-elseifconstruct since if-elseif might require severalcondition evaluations, while the select case onlycontains one condition evaluation. The selectcase construct is analogous to the depreciated com-puted goto.† Also select case constructs conveycontrol logic in clearer fashion and allow for cleanererror handling through the default case.

Performance Considerations of Fortran 95

Throughout the Fortran 95 restructuring of theFun3D solver, several efficiency issues pertaining toadvanced coding constructs were uncovered. Fea-tures such as derived types and modules are ex-tremely attractive for communicating data; however,it was found that current Fortran 95 compilersoften failed to produce performance comparable tothat of conventional Fortran 77 constructs such aspassing data through calling argument lists.

Data Sharing With ModulesAn intermediate restructuring of Fun3D relied al-

most exclusively on the use of Fortran 95 modules.By eliminating virtually every argument list in thesolver, an exceptionally clean code was obtained.∗For example, use aModule, only : aVariable†See

groups.google.com/groups?threadm=9o7uhi%24pus%241%40eising.k-net.dk

for further discussion.

17 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

Table B1 Compilers used in performance study.

Vendor Options Release O/S Platform

Absoft� -O3 -cpu:p6 8.0-1 Linux® 2.4.18 Intel® P3

Compaq® -arch ev67 -fast -O4 -tune ev67 X1.1.1-1684 Linux® 2.4.2 Alpha EV67

HP® -O3 2.4 HP-UX® B.10.20 HP® 9000

IBM® -O5 7 AIX® 3 IBM® 7044

Intel® -O3 -ipo -wK 7.1-008 Linux® 2.4.18 Intel® P3

Lahey-Fujitsu --o2 --nwarn -static --nsav --ntrace --nchk

-x -

6.20a Linux® 2.4.18 Intel® P3

NAG® -O4 -Wc,-malign-double -ieee=full

-unsharedf95

4.2 Linux® 2.4.18 Intel® P3

NA Software -fast 2.2-1 Linux® 2.4.18 Intel® P3

PGI® -fast 4.1-1 Linux® 2.4.18 Intel® P3

SGI® -O2 7.3.1.2m IRIX® 6.5 SGI® R10000

Sun� -fast 6.2-2 SunOS� 5.8 Sun� Blade1000

Table B2 Unformatted disk I/O using 20M integers and 20M reals.

Compiler Assumed size Module Derived type Assumed shape

Absoft� 1.00 1.00 1.03 1.04

Compaq® 1.00 0.98 6.47 0.99

IBM® 1.00 1.03 1.01 1.03

Intel® 1.00 1.16 1.14 1.05

Lahey/Fujitsu 1.00 6.05 5.99 1.04

NAG® 1.00 1.02 1.22 1.03

NA Software 1.00 0.99 1.08 1.01

PGI® 1.00 0.98 1.00 0.98

SGI® 1.00 31.91 31.37 34.80

Sun� 1.00 0.98 1.02 0.98

Table B3 Compute work using 20M integers and 20M reals.

Compiler Assumed size Module Derived type Assumed shape

Absoft� 1.00 1.16 1.84 1.20

Compaq® 1.00 1.40 1.47 1.38

IBM® 1.00 2.76 2.76 2.76

Intel® 1.00 0.97 0.98 0.95

Lahey/Fujitsu 1.00 1.07 1.07 1.02

NAG® Aborted

NA Software 1.00 0.95 1.13 0.92

PGI® 1.00 1.96 1.96 0.94

SGI® 1.00 1.10 1.10 1.07

Sun� 1.00 1.42 1.40 1.07

However, in subsequent testing, this implementationwas shown to be several times slower in executionspeed than the legacy C/Fortran 77 solver. Uponcloser inspection, it was found that the use of mod-ules to communicate large segments of data can beextremely inefficient. To illustrate this degradationin performance, the test code included in Refer-ence43 has been executed on a range of platformsand compilers as listed in Table B1. Here, datais communicated with a file I/O routine, as wellas a routine that performs a large amount of ar-

bitrary floating-point manipulations. In addition toan array A passed through a traditional argumentlist interface, an identical array B is also passed toand from the subroutines through the use of a For-

tran 95 module. For this test, the extent of thearrays is 20 M, a value on the order of that en-countered in typical aerodynamic simulations. Theresults are normalized on the data obtained usingthe argument list model. As can be seen in Ta-ble B2, use of the module construct can incur severepenalties for unformatted disk I/O. The module in-

18 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

terface is over thirty times slower than the datatransferred via a conventional argument list on anSGI®. For floating-point arithmetic, the module in-terface exhibits run times on the order of 20 percenthigher than the computations using data broughtin through an argument list, as shown in Table B3on the facing page. Due to this performance degra-dation, the module construct is employed sparinglyin the Hefss solver as a means to share large datastructures. Only small amounts of data such asfree-stream quantities, algorithmic parameters, andturbulence modeling constants are shared throughmodules.

Derived Types

The baseline C/Fortran 77 solver was also refac-tored to make extensive use of the Fortran 95derived type construct. The derived type is veryattractive in the sense that a number of relatedquantities can be encapsulated in a single variable,yielding relatively short argument lists throughoutthe code. Using this paradigm, variables related tothe computational grid are stored in a grid type;solution-related variables are located in a soln type,and so forth. When a low-level routine requires afundamental piece of data such as the coordinates ofa grid point i, the information can be extracted asgrid%x(i), grid%y(i), and grid%z(i). Arrays ofderived types are also supported under Fortran 95,making the implementation of algorithms such asmultigrid and multiple instances of quantities, suchas boundary groups, straightforward.

As in the case of modules, it was found that theuse of derived types can also incur severe executionpenalties. As shown in the last column of Tables B2and B3 on the preceding page, a similar test to theone described previously has been performed on anarray C transferred as the component of a derivedtype variable. It can be seen in Tables B2 and B3on the facing page that this coding idiom can yieldexecution times more than thirty times slower forunformatted disk I/O and nearly a factor of threeslower for floating-point operations over the argu-ment list model.

The current Hefss solver uses derived types toencapsulate much of its data structures; however,the components of these types required by low-levelroutines are extracted at the calling level and are re-ceived as conventional scalars and arrays in the I/O-and compute-intensive portions of the code. Thismodel allows simple argument lists at the higher lev-els of the code, while maintaining the performance ofthe baseline solver. From a developer’s point of view,derived types are one of the more useful enhance-

ments of Fortran 95 over Fortran 77. Theyallow the developer to string together variables inmeaningful groups and treat them as a single entitywhen desired. The Hefss code uses a number ofderived types. For example, the grid derived typecontains all the information needed for the specifi-cation of the discretized mesh—x,y,z values for eachpoint in space, cell volumes, cell-face normals and ar-eas, connectivity information, and so on. Any of thisinformation is available with the simple constructgrid%variable, e.g., grid%x. Derived types mayalso be concatenated, extending their usefulness.For example, the grid derived type in the Hefss codeencompasses a boundary condition derived type thatcontains all the necessary data to impose boundaryconditions—the physical condition (e.g., solid wall),the locations of points on the boundary, surface nor-mals, and so forth. In addition, the definition of thederived type may be extended at a future date with-out affecting existing code. For example, adding acell-face velocity for moving grid applications wouldinvolve a one-line addition to the type definition andwould be completely transparent to sections of codenot requiring this information.

Assumed-Shape Arrays

As shown in Tables B2 and B3 on the preced-ing page, some compilers treat arguments passed viaassumed-shape arrays as poorly as they did derivedtypes. Assumed-shape arrays can be noncontiguous,and thus interfacing to old Fortran 77 routinesmay require data to be copied to form a contigu-ous data block. These data copies can cause a largeincrease in the total memory required to computea flow solution for some compilers as compared toothers.

Memory Copies

Occasionally it is desirable to bring variables intoa routine via argument lists rather than modules,as demonstrated in Tables B2 and B3 on the facingpage. However, unexpected behavior was detectedon certain platform/compiler combinations when ar-gument lists were combined with low-level moduleuse. In these instances, the variables in the moduleswere not synchronized with the argument list vari-ables. This synchronization issue was resolved whenargument lists were used consistently throughout thesubroutines that needed access to the data. It waseventually surmised that this problem was due tomemory copies made by some compilers during asubroutine call. When that data copy was modified,it was no longer synchronized with the original datastored in the module and accessed with use. Also,

19 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978

on return from the subroutine, the local copy of thedata was used to overwrite the data stored in themodule, possibly erasing any modifications of theoriginal data while the copy existed. This behaviorappears to be very compiler and application specificand very difficult to detect and instrument.

Compilation Errors, Warnings, and InformationThe various compilers listed in Table B3 on

page 18 generally have different sets of constructsthat deem errors or produce a warning or other in-formation. Some of the compilers are generally morelenient or particular than others when it comes tothe constructs that are accepted as valid code forcompilation. The Fortran 95 code base has ben-efited from exposure to a large number of differentcompilers. The coding standard contains guidelinesfor promoting portability. This portability experi-ence was gained by exposure to multiple compilers,which makes it important to build and test on manydifferent architectures/compilers, and which also re-sults in a code base that is very portable.

Compiler MaturityIn addition to the problems discussed with per-

formance, errors have been found in a number ofcompilers. Some versions of the compilers have con-tained errors that have prevented them from suc-cessfully compiling Hefss. Also, compiled code willsometimes suffer run-time errors that are specific tothe compiler or its version. Some compiler vendorshave been very quick to respond to compiler bugreports, and others have ignored our requests for res-olution of these errors.

20 of 20

American Institute of Aeronautics and Astronautics Paper 2003-3978