© 2008 The MITRE Corporation. All rights reserved
ReuseVersus Reinvention **
Mary Ann Malloy, PhD
LFM 2008 Conference
** How Will Formal Methods Deal with Composable Systems?
© 2008 The MITRE Corporation. All rights reserved
3
As a public interest company, MITRE
works in partnership with the U.S. government to address
issues of critical national importance.
© 2008 The MITRE Corporation. All rights reserved
4
Disclaimers
The views, opinions, and conclusions expressed here are those of the presenter and should not be construed as an official position of MITRE or the United States Department of Defense (DoD).
All information presented here is UNCLASSIFIED, technically accurate, contains no critical military technology and is not subject to export controls.
© 2008 The MITRE Corporation. All rights reserved
5
What you WILL NOT take away today
A “shrink-wrapped” solution to DoD’s emergent testing challenges.
© 2008 The MITRE Corporation. All rights reserved
6
What you WILL take away today
An understanding of what “service-orientation” and “composability” mean.
Insight regarding how DoD is trying to build composable systems, but may not be testing them appropriately nor learning from the testing it does do.
Ideas for formal methods / testing investigation paths that may improve the state of composable systems verification & testing.
© 2008 The MITRE Corporation. All rights reserved
7
Overview
Background
Examples
Recent Work
Challenges
Summary
© 2008 The MITRE Corporation. All rights reserved
Who uses services?
9
Answer: YOU do! … and DoD wants to!Answer: YOU do! … and DoD wants to!
OnlineBANKINGBANKING
DIRECTIONSDIRECTIONS
TRAVELTRAVEL
eCOMMERCEeCOMMERCE
NEWS & INFORMATION
NEWS & INFORMATION
© 2008 The MITRE Corporation. All rights reserved
What is a service?
10
Characteristics of services– Modular; composable, much like “lego”
building blocks– Network-accessible– Reusable– Standards-based– Distributed capabilities
“A mechanism to enable access to one or more capabilities, where the access is provided using a prescribed interface and
is exercised consistent with constraints and policies as specified by the service description.”
– DoD Net-Centric Services StrategyMay 2007
© 2008 The MITRE Corporation. All rights reserved
What DoD sees…
11
Ability to sort by type of
incident, date, location, etc.
Ability to sort by type of
incident, date, location, etc.
Listing of bomb-related events between 14 Feb
08 and 15 Feb 08
Listing of bomb-related events between 14 Feb
08 and 15 Feb 08
Worldwide threats and incidents: airport, chemical,
bridge, railway, bombs, etc. It also has links to related
news stories and a searchable database.
Worldwide threats and incidents: airport, chemical,
bridge, railway, bombs, etc. It also has links to related
news stories and a searchable database.
…and wants!
© 2008 The MITRE Corporation. All rights reserved
What is “service-oriented architecture”?
12
“A paradigm for organizing and utilizing distributed capabilities that may be under the control of different
ownership domains.”
– OASIS Reference Model for Service-Oriented Architecture October 2006
An architectural style based on flexibly linked software components that leverage web standards and services– NOT a product– NOT a bunch of web services
© 2008 The MITRE Corporation. All rights reserved
What is composability?
What is a composable system?– one that consists of recombinant atomic behaviors (components) selected and assembled to
satisfy specific [new] processing requirements.
NOTE: Composability is meaningful at many layers of abstraction.
13
Composable solutions – the desired end-state of a full-scale SOA implementation – are the direction DoD, federal stakeholders & commercial enterprises are evolving their automation assets.
Composable solutions – the desired end-state of a full-scale SOA implementation – are the direction DoD, federal stakeholders & commercial enterprises are evolving their automation assets.
A design principle dealing with interrelated components that do not make assumptions about the compositions
that may employ them; they are “fit for the unforeseen.”
– proposed definition
© 2008 The MITRE Corporation. All rights reserved
Testing principles Testing for composability
– Ensure individual processing elements do not make undue assumptions about the composition
Code analysis or inspection for “hidden assumptions” or “out of band” dependencies
14
Testing the composition– Validate the chosen composition of individual elements
performs the desired functions.– The “composition layer” is an additional one that must be
tested separately. A composition can be VALID yet still not do anything USEFUL with
respect to the relevant CONTEXT
© 2008 The MITRE Corporation. All rights reserved
Capability Delivery
Capability Delivery
Enables
Typical DoD approach to testing compositions
16
Drives
Data Needed
Data Needed
Service Implementations
Service Implementations
STARTHERE
Drives
Service NeededService Needed
Capability Demonstration
Capability Demonstration
DECLARE SUCCESS!
Enables
Enables
Community Information Exchange
Vocabulary
Community Information Exchange
Vocabulary
Drives
CapabilityNeed
CapabilityNeed
© 2008 The MITRE Corporation. All rights reserved
Better DoD example: Net-Centric Diplomacy **
General findings from the testing of the NCD initiative of Horizontal fusion:– Many different types of interrelated testing are needed.– Exhaustive testing is impossible.
testing must still be iterative it is time consuming! Operationally specific test cases are needed
– Performance testing must focus on service dependencies vice user interface.
– The number of requests that will cause a web service to fail is far lower than for a web server.
17
“Few realize the complexity that must be taken into account when attempting to quantitatively measure performance
and reliability when dealing with web services.”
– Derik Pack, SPAWAR System Center, 2005
** see http://www.dtic.mil/ndia/2005systems/thursday/pack2.pdf
© 2008 The MITRE Corporation. All rights reserved
Better DoD example: Net-Centric Diplomacy concluded
Testing was conducted until “error thresholds” were reached:– Round trip time (90 sec)– Error (15%)
Specific findings– A mean of 3.06 Connections per Second could be achieved– WSDLs define interfaces, but not valid service use
18
Is this practicable across all of DoD?DoD may need to stand up multiple access points for heavily used
services / compositions; and the “sweet spot” will likely differ in times of war vice times of peace.
Is this practicable across all of DoD?DoD may need to stand up multiple access points for heavily used
services / compositions; and the “sweet spot” will likely differ in times of war vice times of peace.
© 2008 The MITRE Corporation. All rights reserved
What DoD must create to “get there…”
Loosely coupled, relevant, “right-sized” services that can be leveraged across continuously changing processes.
New governance that can deal with complex management of distributed, loosely coupled, dynamically composable services.
A better understanding of maintenance implications:– How long does it take? How will other components or clients be
impacted?– Components with low or unknown MTBF should be highly
accessible and easily replaceable…can this be automated?
Rapidly deployable, virtual, continuous test environment– Examples provided on the next two slides …
20
© 2008 The MITRE Corporation. All rights reserved
…something like ELBA? **
21** see http://www.static.cc.gatech.edu/systems/projects/Elba/pub/200604_NOMS_mulini.pdf
1) Developers provide design-level specifications of model and policy documents (as input to Cauldron) and a test plan (XTBL). 2) Cauldron creates a provisioning and deployment plan for the application. 3) Mulini generates staging plan from the input components referred to from XTBL (dashed arrows). 4) Deployment tools deploy the application, monitoring tools to the staging environment. 5) The staging is executed. 6) Data from monitoring tools is gathered for analysis. 7) After analysis, developers adjust deployment specifications or possibly even policies and repeat the process.
© 2008 The MITRE Corporation. All rights reserved
…or STARSHIP?
Key component of the Electronic Proving Ground Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) tool kit for live distributed test environments.
Provides a “threads-based” composable environment to plan, generate planning documents, verify configuration, initialize, execute, synchronize, monitor, control, and report the status of sequence of activities.
Freely available & customizable to any problem domain.
Complexity may be a barrier.
22
POC: Ms. Janet McDonald
(520) 538-3575 (DSN 879)
PM ITTS IMO – Ft. Huachuca
© 2008 The MITRE Corporation. All rights reserved
What MITRE is doing
c2c-composable-c2-list and Community Share site Composable C2 is a “Grand Challenge Problem” within 2009
MITRE Innovation Program (MIP): – How to build reconfigurable components that can be mashed
together in an agile fashion Visualization and Analysis Info Sharing Interoperability and Integration Resource Management to enable composablility (of people,
organizations, networks, sensors, platforms…) Acquisition and Systems Engineering Collaborative and Distributed C2
Example proposal: Web Service Process Optimization
23
“Our hypothesis is that web service optimization can be realized through machine learning techniques and statistical methods. In this research
we intend to find a computational solution to the problem of creating and maintaining web service processes.”
© 2008 The MITRE Corporation. All rights reserved
What MITRE is doing concluded
Resources for Early and Agile Testing– Recently showed how low-cost simulation games can
create a simple, “good-enough” simulation capability to evaluate new concepts early in development and expose the most challenging issues.
– REACT “Online” A composed testing environment for composed solutions!
a loosely coupled simulation capability delivering dynamic flexibility for “quick look” experiments
24
A series of brainstorming sessions on Composable C2 – “static” vs. “dynamic” composablity viz legacy systems– do services derived from a “proven capability” have lower or
non-existent testing requirement
© 2008 The MITRE Corporation. All rights reserved
26
Practical challenges
Can we “right-size” testing as “fit-for-composition?”– Is composability binary? A sliding scale? When is it [not]
OK to use “lower-rated” components?– Can we characterize the right amount of testing based on
the anticipated longevity of the composition? Other factors?
What metadata must be exposed to assess contextual validity of components in composition?– Should WSDL be enriched? Supplemented?– Can what constitutes valid compositions be
expressed as rules? How narrowly / broadly?
What thresholds / metrics are required? Nice to have?– Performance thresholds? Ongoing component health?
Can we “borrow” ideas from other composability abstractions for applicability here?
© 2008 The MITRE Corporation. All rights reserved
Levels of composability testing?
27
testing
Ris
k (e
.g.,
los
s o
f lif
e)
Can we “rate” the composability of components? Can we “rate” the composability of components?
For a composition that will only be used a few times, can we tolerate higher risk?
For a composition that will only be used a few times, can we tolerate higher risk?
as-is for composable C2
© 2008 The MITRE Corporation. All rights reserved
“Pressure-points” for formalisms
How can the lessons-learned from the past inform the way ahead for extending formal methods to testing & verification of composable systems?
Can we derive principles to compose systems in methodical, rather than ad-hoc ways, that will produce more satisfactory results?
How can we handle partial and incremental specifications?
How can we cope when building a composition with parts that make incompatible assumptions about their mutual interactions?
What kinds of automated checking and analysis can we support?
28
© 2008 The MITRE Corporation. All rights reserved
Take-away points
DoD will continue to deploy composed solutions to realize its SOA vision.
Current testing focuses more on the level of service provided and less on how reliably the capability is delivered or whether it actually meets the need.
Different levels of testing are probably appropriate for different contexts (“static” versus “dynamic,” use frequency, loss-of-life consequences).
Automated environments are needed to test composed solutions targeted for rapid deployment
29
© 2008 The MITRE Corporation. All rights reserved
30
Pointers to more information
www.thedacs.com– Data & Analysis Center for Software: A repository of documents,
tools in research areas including testing and reuse
Search for the latest results on:
composable systems
composabilityweb servicetesting
www.peostri.army.mil/PRODUCTS/STARSHIP/– Starship II homepage
testing composable
composable C2
© 2008 The MITRE Corporation. All rights reserved
33
ACRONYMS
C2 = Command & Control C4ISR = Command, Control, Communications, Computers,
Intelligence, Surveillance and Reconnaissance DoD = Department of Defense MTBF = mean time-between-failures NDC = Net-Centric Diplomacy PEO STRI = Program Executive Office for Simulation, Training
& Instrumentation SOA = service-oriented architecture WSDL = web services descriptive language
© 2008 The MITRE Corporation. All rights reserved
Observations about compositions
Solutions are built from primitive and composite components and connectors.
Components and connectors can be described by interface and protocol specifications.
Common patterns provide abstractions we may be able to exploit in design, development, analysis and testing.
To delivery meaningful capability, the components must be composable regarding their underlying ideas
35
Tec
hn
ical
rea
lmS
eman
tic
rea
lm
© 2008 The MITRE Corporation. All rights reserved
36
“Lines of Evolution” vision for DoD systems
A single system with a non-
flexible hierarchical
structure
A system consisting of several independently
functional but integrated components
A capability is realized
through a pre-defined
orchestration of services
Reusable, “mobile” services
Services are orchestrated on an ad-hoc basis
to deliver a capability…and then disappear
1
2
3
4
5
© 2008 The MITRE Corporation. All rights reserved
Another view: stages of SOA adoption
37
1 2 3 4 5 6Business Process Understanding: How is the work done?
IT Assessment: What IT assets exist supporting the business process?
SOA Design/ Determination: What should be a service?
SOA Enablement (Java EE, .NET, federated data services): How will application and data services be developed and deployed?
Infrastructure (ESB, Registry, Management Governance: How will services, application, people interact and communicate?
Process Orchestration/ Composition: How will business processes and rules be developed and deployed?
** Mark Driver, Optimizing Open Source and SOA Strategies, Gartner Application Architecture, Development & Integration Summit 2007, http://www.gartner.com/it/page.jsp?id=506878&tab=overview
DoD is lurking around here
DoD is lurking around here