1 An exploratory investigation on the Invasiveness of
Environmental Modeling Frameworks 18 th IMACS World Congress,
MODSIM09 Cairns, Australia July 2009 Wes J. Lloyd, Olaf David,
James C. Ascough, Ken Rojas, Jack Carlson, George Leavesley, Peter
Krause, Tim Green, Lajpat Ahuja
Slide 2
2 Framework Invasiveness Coupling between application code and
framework code Coupling between application code and framework code
Use of framework functions/methods Use of framework
functions/methods Use of framework specific data types Use of
framework specific data types Implementation of framework
interfaces Implementation of framework interfaces Extension of
framework classes Extension of framework classes Import/Include of
framework libraries Import/Include of framework libraries
Slide 3
3 Framework Invasiveness Object-oriented coupling correlates
inversely with software quality (Briand, 2000; Basil, 1996).
Object-oriented coupling correlates inversely with software quality
(Briand, 2000; Basil, 1996). Framework invasiveness may impact
Framework invasiveness may impact Understandability
Understandability Maintainability Maintainability Upgrading the
framework Upgrading the framework Ease of bug defects / Feature
Enhancements Ease of bug defects / Feature Enhancements Portability
Portability Reusability Reusability
Slide 4
4 Framework Design Classification Heavyweight
FrameworksLightweight Frameworks Overloads/wraps native language
data types Large programming interface (API) Components
tightly-coupled / bound to framework at compile time May use many
external libraries Uses native language data types Small
programming interface (API) Components loosely-coupled, largely
framework independent Components bound to framework
dynamically:language annotations, dependency injection, IoC design
pattern
Slide 5
Research Questions How does framework design (lightweight vs.
heavyweight) impact invasiveness? How does framework design
(lightweight vs. heavyweight) impact invasiveness? How does
framework to application invasiveness impact Software Quality? in
terms of: How does framework to application invasiveness impact
Software Quality? in terms of: Maintainability Maintainability
Understandability Understandability Portability Portability
Reusability Reusability
Slide 6
6 Measuring Invasiveness Are Object Oriented Coupling Measures
useful? Are Object Oriented Coupling Measures useful? Coupling
Between Object Classes (CBO) Coupling Between Object Classes (CBO)
Efferent Coupling / Fan Out Efferent Coupling / Fan Out Afferent
Coupling / Fan In Afferent Coupling / Fan In Response for a Class
(RFC) Response for a Class (RFC) Message Passage Coupling (MPC)
Message Passage Coupling (MPC) Framework Invasiveness measures
needed Framework Invasiveness measures needed
Slide 7
7 Invasiveness Metrics Count # Used, # of Uses Count # Used, #
of Uses Framework Data Types (FDT) Framework Data Types (FDT)
Framework Functions (FF) Framework Functions (FF) Framework
Dependent Lines of Code (FDLOC) Framework Dependent Lines of Code
(FDLOC) Three scalings Three scalings Raw count Raw count Per 1000
LOC (kloc) Per 1000 LOC (kloc) As a % of all data types used As a %
of all data types used
Slide 8
8 Empirical Study Environmental Modeling Frameworks
Environmental Modeling Frameworks Support aggregation of models
into classes (components) Support aggregation of models into
classes (components) Component interaction/communication Component
interaction/communication Time/spatial data looping Time/spatial
data looping Regridding arrays and spatial data Regridding arrays
and spatial data Multithreading/multiprocessor support
Multithreading/multiprocessor support Cross language
interoperability Cross language interoperability
Slide 9
9 Environmental Modeling Frameworks CCA 0.6.6: Common Component
Architecture - Java CCA 0.6.6: Common Component Architecture - Java
ESMF-C/Fortran 3.1.1: Earth Science Modeling Framework
ESMF-C/Fortran 3.1.1: Earth Science Modeling Framework OpenMI 1.4:
Open Modeling Interface - Java OpenMI 1.4: Open Modeling Interface
- Java OMS 2.2: Object Modeling System - Java OMS 2.2: Object
Modeling System - Java OMS 3.0: Object Modeling System- Java OMS
3.0: Object Modeling System- Java
Slide 10
10 Modeling Application: Thornthwaite Modeling Application:
Thornthwaite Thornthwaite Water balance model Thornthwaite Water
balance model Models allocation of water among components of
hydrological system Models allocation of water among components of
hydrological system Model Model 8 Components 8 Components Climate,
Daylen, HamonET, Snow, Soil moisture, Runoff, Output, Controller
Climate, Daylen, HamonET, Snow, Soil moisture, Runoff, Output,
Controller FORTRAN Implementation = 244 LOC FORTRAN Implementation
= 244 LOC
12 Modeling Application: Thornthwaite All implementations
produce identical numeric output No language specific output
formatting Source code repository:
http://svn.javaforge.com/svn/invasive/trunk/
http://svn.javaforge.com/svn/invasive/trunk/
Slide 13
13 Analysis Tools SLOCCOUNT LOC for FORTRAN, C, C++, Java LOC
for FORTRAN, C, C++, Java Understand 2.0 Analyst metrics for
FORTRAN, C, C++, Java metrics for FORTRAN, C, C++, Java Custom tool
Parsed Understand 2.0 function and data type usage reports to
provide data for FF and FDT usage measures Parsed Understand 2.0
function and data type usage reports to provide data for FF and FDT
usage measures
Slide 14
14 Model implementations
Slide 15
15 Framework Dependent Code
Slide 16
16 Framework Data Types Used
Slide 17
17 Framework Data Type Uses
Slide 18
18 Framework Functions Used
Slide 19
19 Framework Function Uses
Slide 20
20 Invasiveness Measures Combine measures to generate an
overall invasiveness ranking Combine measures to generate an
overall invasiveness ranking Invasiveness 1: raw counts
Invasiveness 1: raw counts FDLOC, FDT Used, FDT Uses, FF Used, FF
Uses FDLOC, FDT Used, FDT Uses, FF Used, FF Uses Invasiveness 2:
framework usage density Invasiveness 2: framework usage density
Framework to non-framework data type and function usage Framework
to non-framework data type and function usage %FDLOC, %FDT Used,
%FDT Uses, %FF Used, %FF Uses %FDLOC, %FDT Used, %FDT Uses, %FF
Used, %FF Uses Invasiveness 3: code density Invasiveness 3: code
density Framework data type and function usage per kloc Framework
data type and function usage per kloc FDLOC/kloc, FDT Used/kloc,
FDT Uses/kloc, FF Used/kloc, FF Uses/kloc FDLOC/kloc, FDT
Used/kloc, FDT Uses/kloc, FF Used/kloc, FF Uses/kloc
Slide 21
21 For ranking Invasiveness (FDT Used/kloc + FDT Uses/kloc)/2 +
(FF Used/kloc + FF Uses/kloc)/2 + % FDLOC To generate invasiveness:
To generate invasiveness: Calculate averages, standard deviations
for each metric, for each model implementation Calculate averages,
standard deviations for each metric, for each model implementation
Use the number of standard deviations away from average in place of
metric value Use the number of standard deviations away from
average in place of metric value Sum (or average) the standard
deviations Sum (or average) the standard deviations Larger values
indicate more invasive implementations when compared with others in
the set Larger values indicate more invasive implementations when
compared with others in the set
Slide 22
22 Invasiveness rankings
Slide 23
23 Invasiveness Measure Independence H0: There is no
relationship between invasiveness measures H0: There is no
relationship between invasiveness measures 120 possible
relationships 120 possible relationships 8 significant (pearson) 8
significant (pearson) multiple r>.811, df=4, p.811, df=4, p.811,
df=4, p.811, df=4, p
24 Invasiveness Measure Dependence Only six relations cross
measure categories Only six relations cross measure categories Both
Pearson & Spearman Rank Both Pearson & Spearman Rank FDLOC
-> FF Used FDLOC -> FF Used FDLOC -> FF Uses FDLOC ->
FF Uses %FDLOC -> %FF Uses %FDLOC -> %FF Uses %FF Uses ->
FDLOC/kloc %FF Uses -> FDLOC/kloc Pearson Pearson FDT Used ->
FF Used FDT Used -> FF Used Spearman Rank Spearman Rank %FDT
Uses -> %FF Used %FDT Uses -> %FF Used
Slide 25
25 Invasiveness and size (LOC) H 0 : There is no relationship
between invasiveness measures and LOC H 0 : There is no
relationship between invasiveness measures and LOC Significant
correlations Significant correlations FDLOC, FDT Used, FF Used
-> LOC FDLOC, FDT Used, FF Used -> LOC Raw values seem to
correlate with LOC, others do not. Raw values seem to correlate
with LOC, others do not.
Slide 26
26 Invasiveness and Complexity H 0 : There is no relationship
between invasiveness measures and Cyclomatic Complexity H 0 : There
is no relationship between invasiveness measures and Cyclomatic
Complexity Significant correlations Significant correlations FF
Used/KLOC -> Avg CC/method FF Used/KLOC -> Avg CC/method
FDLOC, FDT Used, FF Used -> Total CC FDLOC, FDT Used, FF Used
-> Total CC Cyclomatic complexity and invasiveness do not appear
to be related Cyclomatic complexity and invasiveness do not appear
to be related
Slide 27
27 Invasiveness and Coupling H 0 : There is no relationship
between application to framework coupling measures and Object
Oriented Coupling Measures (CBO, Fan-In, Fan-Out) H 0 : There is no
relationship between application to framework coupling measures and
Object Oriented Coupling Measures (CBO, Fan-In, Fan-Out)
Significant correlations Significant correlations FDLOC, FDT Used,
%FF Used -> Total Fan In FDLOC, FDT Used, %FF Used -> Total
Fan In FDLOC, FDT Used -> Total Fan Out FDLOC, FDT Used ->
Total Fan Out CBO: only two systems had a measured value CBO: only
two systems had a measured value Raw values seem to correlate with
total fan- in/fan-out coupling, others do not. Raw values seem to
correlate with total fan- in/fan-out coupling, others do not.
Slide 28
28 Invasiveness and Indirect Measures of Software Quality
Chidamber and Kemerer OO metrics Chidamber and Kemerer OO metrics
WMC: Weighted methods per class WMC: Weighted methods per class
CBO: Coupling between object classes CBO: Coupling between object
classes RFC: Response for a class RFC: Response for a class LCOM:
Lack of cohesion in methods LCOM: Lack of cohesion in methods
Metrics only collected for OO systems Metrics only collected for OO
systems n=4, df=2, very limited sample size! n=4, df=2, very
limited sample size!
Slide 29
29 Invasiveness and Indirect Measures of Software Quality H 0 :
The quantity of application to framework coupling/invasiveness is
not related to indirect measurements of software quality. H 0 : The
quantity of application to framework coupling/invasiveness is not
related to indirect measurements of software quality. df=2 requires
multiple r>=.950! df=2 requires multiple r>=.950!
Slide 30
30 Experiment miscellaneous details CCA implementation: an
outlier CCA implementation: an outlier Large quantity of
automatically generated boilerplate code Large quantity of
automatically generated boilerplate code To normalize size ignored
files not touched by developer To normalize size ignored files not
touched by developer Reduces LOC from 62809 (all) to 9914 (java
only), to 1635 (user java only) Reduces LOC from 62809 (all) to
9914 (java only), to 1635 (user java only) NGMF NGMF Did not count
lines of code with just a Java annotation as a framework dependent
line of code (FDLOC) Did not count lines of code with just a Java
annotation as a framework dependent line of code (FDLOC) Treating
annotations like comments Treating annotations like comments
Annotations not used in other frameworks Annotations not used in
other frameworks Annotations are easily removed/ignored Annotations
are easily removed/ignored ESMF C ESMF C Implementation used global
data due to incomplete framework support Implementation used global
data due to incomplete framework support Model functions not
counted as framework functions Model functions not counted as
framework functions Model datatypes not counted as framework
datatypes Model datatypes not counted as framework datatypes
Fortran datatype usage counted manually without tool support
Fortran datatype usage counted manually without tool support
Slide 31
31 External threats to validity Thornthwaite is a simple model
that does not fully exercise frameworks Thornthwaite is a simple
model that does not fully exercise frameworks Model developer was
new to developing in frameworks. Two implementations were already
coded. Model developer was new to developing in frameworks. Two
implementations were already coded. Implementation languages
varied, therefore metrics collection techniques varied
Implementation languages varied, therefore metrics collection
techniques varied Not all implementations were in an OO language
Not all implementations were in an OO language
Slide 32
32 Other Limitations Limited experimental power Limited
experimental power Framework implementations (n=6, df=4) Framework
implementations (n=6, df=4) Some measures were assessed manually
without tool support Some measures were assessed manually without
tool support
Slide 33
33 Summary Unique comparison study performed Unique comparison
study performed Thornthwaite scientific model implemented 8 times
in 5 frameworks, 4 languages Thornthwaite scientific model
implemented 8 times in 5 frameworks, 4 languages Invasiveness
metrics proposed, applied, and evaluated Invasiveness metrics
proposed, applied, and evaluated Measures used to rank
framework-based implementations Measures used to rank
framework-based implementations Measures compared with existing
software metrics: LOC, complexity, OO coupling Measures compared
with existing software metrics: LOC, complexity, OO coupling
Measures compared with indirect measures of software quality
Measures compared with indirect measures of software quality
Slide 34
34 Conclusions Individual invasiveness measures correlated
significantly with each other only slightly more than random chance
Individual invasiveness measures correlated significantly with each
other only slightly more than random chance Raw invasiveness metric
values did correlate somewhat with application size (LOC) Raw
invasiveness metric values did correlate somewhat with application
size (LOC) There was no significant relationship between
invasiveness and complexity There was no significant relationship
between invasiveness and complexity Raw invasiveness metric values
did correlate somewhat with total fan-in / total fan-out coupling
Raw invasiveness metric values did correlate somewhat with total
fan-in / total fan-out coupling
Slide 35
35 Future Investigations - 1 Case Study: Broad Investigation of
Invasiveness Case Study: Broad Investigation of Invasiveness
Compare Open Source framework based applications Compare Open
Source framework based applications Several applications
implemented in the same framework Several applications implemented
in the same framework Compare frameworks from particular domains
Compare frameworks from particular domains What general trends are
seen for invasiveness? What general trends are seen for
invasiveness? How does invasiveness relative to other measures?
(LOC, Complexity, OO Coupling) How does invasiveness relative to
other measures? (LOC, Complexity, OO Coupling)
Slide 36
36 Future Investigations - 2 Invasiveness study with a larger
scientific model, or several models Invasiveness study with a
larger scientific model, or several models Implement one or more
models which more fully exercise modeling frameworks Implement one
or more models which more fully exercise modeling frameworks
Compare results with initial study Compare results with initial
study Additional models add more data for better experimental power
(>N) Additional models add more data for better experimental
power (>N)
Slide 37
37 Future Investigations - 3 Invasiveness study in another
domain Invasiveness study in another domain Implement a comparison
application in multiple frameworks in a different domain such as
web application frameworks (struts, tapestry, etc.) Implement a
comparison application in multiple frameworks in a different domain
such as web application frameworks (struts, tapestry, etc.) What
are the implications of invasiveness in other application domains?
What are the implications of invasiveness in other application
domains?
Slide 38
38 Future Investigations - 4 Maintainability controlled
experiment Maintainability controlled experiment For one or more
systems implemented in different frameworks perform (5+)
maintenance tasks For one or more systems implemented in different
frameworks perform (5+) maintenance tasks Bug fixes Bug fixes
Enhancement Requests Enhancement Requests Implement changes in each
implementation Implement changes in each implementation Questions
with Respect to Invasiveness: Questions with Respect to
Invasiveness: Were changes functionally achievable? Were changes
functionally achievable? What is the effort (time/LOC) to make
changes? What is the effort (time/LOC) to make changes? What is the
quality of fixes/enhancement implementations? What is the quality
of fixes/enhancement implementations?
Slide 39
39 Future Investigations - 5 Maintainability controlled
experiment II Maintainability controlled experiment II For one or
more systems implemented in different frameworks assign developers
new to the application maintenance tasks For one or more systems
implemented in different frameworks assign developers new to the
application maintenance tasks Bug fixes Bug fixes Enhancement
Requests Enhancement Requests Questions with Respect to
Invasiveness: Questions with Respect to Invasiveness: Were
developers able to implement changes in a fixed period of time?
Were developers able to implement changes in a fixed period of
time? What effort (time/LOC) was required to make changes? What
effort (time/LOC) was required to make changes? What is the quality
of fixes/enhancement implementations? What is the quality of
fixes/enhancement implementations?
Slide 40
40 Future Investigations - 6 Study Application Maintainability
when frameworks inadvertently change Study Application
Maintainability when frameworks inadvertently change Make
random/planned framework changes Make random/planned framework
changes Update/correct application code Update/correct application
code Questions with Respect to Invasiveness: Questions with Respect
to Invasiveness: How does invasiveness relate to number of
application errors? How does invasiveness relate to number of
application errors? What effort (time/LOC) was required to update
applications? What effort (time/LOC) was required to update
applications? Did framework changes enhance the applications? Did
framework changes enhance the applications?