1 An exploratory investigation on the Invasiveness of Environmental Modeling Frameworks 18 th IMACS World Congress, MODSIM09 Cairns, Australia July 2009

1 An exploratory investigation on the Invasiveness of Environmental Modeling Frameworks 18 th IMACS World Congress, MODSIM09 Cairns, Australia July 2009 Wes J. Lloyd, Olaf David, James C. Ascough, Ken Rojas, Jack Carlson, George Leavesley, Peter Krause, Tim Green, Lajpat Ahuja

2 Framework Invasiveness Coupling between application code and framework code Coupling between application code and framework code Use of framework functions/methods Use of framework functions/methods Use of framework specific data types Use of framework specific data types Implementation of framework interfaces Implementation of framework interfaces Extension of framework classes Extension of framework classes Import/Include of framework libraries Import/Include of framework libraries

3 Framework Invasiveness Object-oriented coupling correlates inversely with software quality (Briand, 2000; Basil, 1996). Object-oriented coupling correlates inversely with software quality (Briand, 2000; Basil, 1996). Framework invasiveness may impact Framework invasiveness may impact Understandability Understandability Maintainability Maintainability Upgrading the framework Upgrading the framework Ease of bug defects / Feature Enhancements Ease of bug defects / Feature Enhancements Portability Portability Reusability Reusability

4 Framework Design Classification Heavyweight FrameworksLightweight Frameworks Overloads/wraps native language data types Large programming interface (API) Components tightly-coupled / bound to framework at compile time May use many external libraries Uses native language data types Small programming interface (API) Components loosely-coupled, largely framework independent Components bound to framework dynamically:language annotations, dependency injection, IoC design pattern

Research Questions How does framework design (lightweight vs. heavyweight) impact invasiveness? How does framework design (lightweight vs. heavyweight) impact invasiveness? How does framework to application invasiveness impact Software Quality? in terms of: How does framework to application invasiveness impact Software Quality? in terms of: Maintainability Maintainability Understandability Understandability Portability Portability Reusability Reusability

6 Measuring Invasiveness Are Object Oriented Coupling Measures useful? Are Object Oriented Coupling Measures useful? Coupling Between Object Classes (CBO) Coupling Between Object Classes (CBO) Efferent Coupling / Fan Out Efferent Coupling / Fan Out Afferent Coupling / Fan In Afferent Coupling / Fan In Response for a Class (RFC) Response for a Class (RFC) Message Passage Coupling (MPC) Message Passage Coupling (MPC) Framework Invasiveness measures needed Framework Invasiveness measures needed

7 Invasiveness Metrics Count # Used, # of Uses Count # Used, # of Uses Framework Data Types (FDT) Framework Data Types (FDT) Framework Functions (FF) Framework Functions (FF) Framework Dependent Lines of Code (FDLOC) Framework Dependent Lines of Code (FDLOC) Three scalings Three scalings Raw count Raw count Per 1000 LOC (kloc) Per 1000 LOC (kloc) As a % of all data types used As a % of all data types used

8 Empirical Study Environmental Modeling Frameworks Environmental Modeling Frameworks Support aggregation of models into classes (components) Support aggregation of models into classes (components) Component interaction/communication Component interaction/communication Time/spatial data looping Time/spatial data looping Regridding arrays and spatial data Regridding arrays and spatial data Multithreading/multiprocessor support Multithreading/multiprocessor support Cross language interoperability Cross language interoperability

9 Environmental Modeling Frameworks CCA 0.6.6: Common Component Architecture - Java CCA 0.6.6: Common Component Architecture - Java ESMF-C/Fortran 3.1.1: Earth Science Modeling Framework ESMF-C/Fortran 3.1.1: Earth Science Modeling Framework OpenMI 1.4: Open Modeling Interface - Java OpenMI 1.4: Open Modeling Interface - Java OMS 2.2: Object Modeling System - Java OMS 2.2: Object Modeling System - Java OMS 3.0: Object Modeling System- Java OMS 3.0: Object Modeling System- Java

10 Modeling Application: Thornthwaite Modeling Application: Thornthwaite Thornthwaite Water balance model Thornthwaite Water balance model Models allocation of water among components of hydrological system Models allocation of water among components of hydrological system Model Model 8 Components 8 Components Climate, Daylen, HamonET, Snow, Soil moisture, Runoff, Output, Controller Climate, Daylen, HamonET, Snow, Soil moisture, Runoff, Output, Controller FORTRAN Implementation = 244 LOC FORTRAN Implementation = 244 LOC

11 Modeling Application: Thornthwaite Eight implementations Eight implementations

12 Modeling Application: Thornthwaite All implementations produce identical numeric output No language specific output formatting Source code repository: http://svn.javaforge.com/svn/invasive/trunk/ http://svn.javaforge.com/svn/invasive/trunk/

13 Analysis Tools SLOCCOUNT LOC for FORTRAN, C, C++, Java LOC for FORTRAN, C, C++, Java Understand 2.0 Analyst metrics for FORTRAN, C, C++, Java metrics for FORTRAN, C, C++, Java Custom tool Parsed Understand 2.0 function and data type usage reports to provide data for FF and FDT usage measures Parsed Understand 2.0 function and data type usage reports to provide data for FF and FDT usage measures

14 Model implementations

15 Framework Dependent Code

16 Framework Data Types Used

17 Framework Data Type Uses

18 Framework Functions Used

19 Framework Function Uses

20 Invasiveness Measures Combine measures to generate an overall invasiveness ranking Combine measures to generate an overall invasiveness ranking Invasiveness 1: raw counts Invasiveness 1: raw counts FDLOC, FDT Used, FDT Uses, FF Used, FF Uses FDLOC, FDT Used, FDT Uses, FF Used, FF Uses Invasiveness 2: framework usage density Invasiveness 2: framework usage density Framework to non-framework data type and function usage Framework to non-framework data type and function usage %FDLOC, %FDT Used, %FDT Uses, %FF Used, %FF Uses %FDLOC, %FDT Used, %FDT Uses, %FF Used, %FF Uses Invasiveness 3: code density Invasiveness 3: code density Framework data type and function usage per kloc Framework data type and function usage per kloc FDLOC/kloc, FDT Used/kloc, FDT Uses/kloc, FF Used/kloc, FF Uses/kloc FDLOC/kloc, FDT Used/kloc, FDT Uses/kloc, FF Used/kloc, FF Uses/kloc

21 For ranking Invasiveness (FDT Used/kloc + FDT Uses/kloc)/2 + (FF Used/kloc + FF Uses/kloc)/2 + % FDLOC To generate invasiveness: To generate invasiveness: Calculate averages, standard deviations for each metric, for each model implementation Calculate averages, standard deviations for each metric, for each model implementation Use the number of standard deviations away from average in place of metric value Use the number of standard deviations away from average in place of metric value Sum (or average) the standard deviations Sum (or average) the standard deviations Larger values indicate more invasive implementations when compared with others in the set Larger values indicate more invasive implementations when compared with others in the set

22 Invasiveness rankings

23 Invasiveness Measure Independence H0: There is no relationship between invasiveness measures H0: There is no relationship between invasiveness measures 120 possible relationships 120 possible relationships 8 significant (pearson) 8 significant (pearson) multiple r>.811, df=4, p.811, df=4, p.811, df=4, p.811, df=4, p

24 Invasiveness Measure Dependence Only six relations cross measure categories Only six relations cross measure categories Both Pearson & Spearman Rank Both Pearson & Spearman Rank FDLOC -> FF Used FDLOC -> FF Used FDLOC -> FF Uses FDLOC -> FF Uses %FDLOC -> %FF Uses %FDLOC -> %FF Uses %FF Uses -> FDLOC/kloc %FF Uses -> FDLOC/kloc Pearson Pearson FDT Used -> FF Used FDT Used -> FF Used Spearman Rank Spearman Rank %FDT Uses -> %FF Used %FDT Uses -> %FF Used

25 Invasiveness and size (LOC) H 0 : There is no relationship between invasiveness measures and LOC H 0 : There is no relationship between invasiveness measures and LOC Significant correlations Significant correlations FDLOC, FDT Used, FF Used -> LOC FDLOC, FDT Used, FF Used -> LOC Raw values seem to correlate with LOC, others do not. Raw values seem to correlate with LOC, others do not.

26 Invasiveness and Complexity H 0 : There is no relationship between invasiveness measures and Cyclomatic Complexity H 0 : There is no relationship between invasiveness measures and Cyclomatic Complexity Significant correlations Significant correlations FF Used/KLOC -> Avg CC/method FF Used/KLOC -> Avg CC/method FDLOC, FDT Used, FF Used -> Total CC FDLOC, FDT Used, FF Used -> Total CC Cyclomatic complexity and invasiveness do not appear to be related Cyclomatic complexity and invasiveness do not appear to be related

27 Invasiveness and Coupling H 0 : There is no relationship between application to framework coupling measures and Object Oriented Coupling Measures (CBO, Fan-In, Fan-Out) H 0 : There is no relationship between application to framework coupling measures and Object Oriented Coupling Measures (CBO, Fan-In, Fan-Out) Significant correlations Significant correlations FDLOC, FDT Used, %FF Used -> Total Fan In FDLOC, FDT Used, %FF Used -> Total Fan In FDLOC, FDT Used -> Total Fan Out FDLOC, FDT Used -> Total Fan Out CBO: only two systems had a measured value CBO: only two systems had a measured value Raw values seem to correlate with total fan- in/fan-out coupling, others do not. Raw values seem to correlate with total fan- in/fan-out coupling, others do not.

28 Invasiveness and Indirect Measures of Software Quality Chidamber and Kemerer OO metrics Chidamber and Kemerer OO metrics WMC: Weighted methods per class WMC: Weighted methods per class CBO: Coupling between object classes CBO: Coupling between object classes RFC: Response for a class RFC: Response for a class LCOM: Lack of cohesion in methods LCOM: Lack of cohesion in methods Metrics only collected for OO systems Metrics only collected for OO systems n=4, df=2, very limited sample size! n=4, df=2, very limited sample size!

29 Invasiveness and Indirect Measures of Software Quality H 0 : The quantity of application to framework coupling/invasiveness is not related to indirect measurements of software quality. H 0 : The quantity of application to framework coupling/invasiveness is not related to indirect measurements of software quality. df=2 requires multiple r>=.950! df=2 requires multiple r>=.950!

30 Experiment miscellaneous details CCA implementation: an outlier CCA implementation: an outlier Large quantity of automatically generated boilerplate code Large quantity of automatically generated boilerplate code To normalize size ignored files not touched by developer To normalize size ignored files not touched by developer Reduces LOC from 62809 (all) to 9914 (java only), to 1635 (user java only) Reduces LOC from 62809 (all) to 9914 (java only), to 1635 (user java only) NGMF NGMF Did not count lines of code with just a Java annotation as a framework dependent line of code (FDLOC) Did not count lines of code with just a Java annotation as a framework dependent line of code (FDLOC) Treating annotations like comments Treating annotations like comments Annotations not used in other frameworks Annotations not used in other frameworks Annotations are easily removed/ignored Annotations are easily removed/ignored ESMF C ESMF C Implementation used global data due to incomplete framework support Implementation used global data due to incomplete framework support Model functions not counted as framework functions Model functions not counted as framework functions Model datatypes not counted as framework datatypes Model datatypes not counted as framework datatypes Fortran datatype usage counted manually without tool support Fortran datatype usage counted manually without tool support

31 External threats to validity Thornthwaite is a simple model that does not fully exercise frameworks Thornthwaite is a simple model that does not fully exercise frameworks Model developer was new to developing in frameworks. Two implementations were already coded. Model developer was new to developing in frameworks. Two implementations were already coded. Implementation languages varied, therefore metrics collection techniques varied Implementation languages varied, therefore metrics collection techniques varied Not all implementations were in an OO language Not all implementations were in an OO language

32 Other Limitations Limited experimental power Limited experimental power Framework implementations (n=6, df=4) Framework implementations (n=6, df=4) Some measures were assessed manually without tool support Some measures were assessed manually without tool support

33 Summary Unique comparison study performed Unique comparison study performed Thornthwaite scientific model implemented 8 times in 5 frameworks, 4 languages Thornthwaite scientific model implemented 8 times in 5 frameworks, 4 languages Invasiveness metrics proposed, applied, and evaluated Invasiveness metrics proposed, applied, and evaluated Measures used to rank framework-based implementations Measures used to rank framework-based implementations Measures compared with existing software metrics: LOC, complexity, OO coupling Measures compared with existing software metrics: LOC, complexity, OO coupling Measures compared with indirect measures of software quality Measures compared with indirect measures of software quality

34 Conclusions Individual invasiveness measures correlated significantly with each other only slightly more than random chance Individual invasiveness measures correlated significantly with each other only slightly more than random chance Raw invasiveness metric values did correlate somewhat with application size (LOC) Raw invasiveness metric values did correlate somewhat with application size (LOC) There was no significant relationship between invasiveness and complexity There was no significant relationship between invasiveness and complexity Raw invasiveness metric values did correlate somewhat with total fan-in / total fan-out coupling Raw invasiveness metric values did correlate somewhat with total fan-in / total fan-out coupling

35 Future Investigations - 1 Case Study: Broad Investigation of Invasiveness Case Study: Broad Investigation of Invasiveness Compare Open Source framework based applications Compare Open Source framework based applications Several applications implemented in the same framework Several applications implemented in the same framework Compare frameworks from particular domains Compare frameworks from particular domains What general trends are seen for invasiveness? What general trends are seen for invasiveness? How does invasiveness relative to other measures? (LOC, Complexity, OO Coupling) How does invasiveness relative to other measures? (LOC, Complexity, OO Coupling)

36 Future Investigations - 2 Invasiveness study with a larger scientific model, or several models Invasiveness study with a larger scientific model, or several models Implement one or more models which more fully exercise modeling frameworks Implement one or more models which more fully exercise modeling frameworks Compare results with initial study Compare results with initial study Additional models add more data for better experimental power (>N) Additional models add more data for better experimental power (>N)

37 Future Investigations - 3 Invasiveness study in another domain Invasiveness study in another domain Implement a comparison application in multiple frameworks in a different domain such as web application frameworks (struts, tapestry, etc.) Implement a comparison application in multiple frameworks in a different domain such as web application frameworks (struts, tapestry, etc.) What are the implications of invasiveness in other application domains? What are the implications of invasiveness in other application domains?

38 Future Investigations - 4 Maintainability controlled experiment Maintainability controlled experiment For one or more systems implemented in different frameworks perform (5+) maintenance tasks For one or more systems implemented in different frameworks perform (5+) maintenance tasks Bug fixes Bug fixes Enhancement Requests Enhancement Requests Implement changes in each implementation Implement changes in each implementation Questions with Respect to Invasiveness: Questions with Respect to Invasiveness: Were changes functionally achievable? Were changes functionally achievable? What is the effort (time/LOC) to make changes? What is the effort (time/LOC) to make changes? What is the quality of fixes/enhancement implementations? What is the quality of fixes/enhancement implementations?

39 Future Investigations - 5 Maintainability controlled experiment II Maintainability controlled experiment II For one or more systems implemented in different frameworks assign developers new to the application maintenance tasks For one or more systems implemented in different frameworks assign developers new to the application maintenance tasks Bug fixes Bug fixes Enhancement Requests Enhancement Requests Questions with Respect to Invasiveness: Questions with Respect to Invasiveness: Were developers able to implement changes in a fixed period of time? Were developers able to implement changes in a fixed period of time? What effort (time/LOC) was required to make changes? What effort (time/LOC) was required to make changes? What is the quality of fixes/enhancement implementations? What is the quality of fixes/enhancement implementations?

40 Future Investigations - 6 Study Application Maintainability when frameworks inadvertently change Study Application Maintainability when frameworks inadvertently change Make random/planned framework changes Make random/planned framework changes Update/correct application code Update/correct application code Questions with Respect to Invasiveness: Questions with Respect to Invasiveness: How does invasiveness relate to number of application errors? How does invasiveness relate to number of application errors? What effort (time/LOC) was required to update applications? What effort (time/LOC) was required to update applications? Did framework changes enhance the applications? Did framework changes enhance the applications?

41 Questions

Documents

1 An exploratory investigation on the Invasiveness of Environmental Modeling Frameworks 18 th IMACS World Congress, MODSIM09 Cairns, Australia July 2009