Project PresentationDocument Optimization

11 May 2007

Team members: Chris CatalanoChun-Yu

ChangChris JosonDavid Matthes

Sponsors:Huron Consulting GroupAerospace Corporation

EDD Background

• Electronic Data Discovery (EDD) is the systematic collection, processing and review of electronic files to support the litigation process.

• EDD is used in:

– Alleged stock-back dating

– Government reviews of mergers and acquisitions

– Other dirty deals e.g. blackmail, fraud, embezzlement

• The current processing system was designed for component flexibility and variability.

• The market place is shifting to an environment that holds speed and automation paramount.

Project Objectives

• Evaluate the current EDD system against two alternatives.• Client: Huron Consulting Group

• Evaluate SysML as an effective modeling language for systems engineering.• Client: Aerospace Corporation

Approach

• Modeled and compared three EDD systems in SysML.

• Evaluated the EDD systems from a capital budgeting perspective

• Evaluated quantitatively our experience with SysML.

Agenda•Approach - SysML Model•Analysis - Trade Study•Evaluation - SysML Usability

Approach - SysML Model

SysML is a modeling language for: •Defining systems•Analyzing systems•Communicating different system viewpoints

EDD Team captured the following viewpoints:•Requirements Diagram•Use Case Diagram•Block Definition Diagram•Activity Diagram

Requirements Diagram

•Captures EDD requirement hierarchy•Provides traceability to EDD components

Use Case Diagram

•Describes EDD contextual relationships•Show interactions between entities and the EDD System

Block Definition Diagram

Describes the characteristics

Describes Behaviors

Describes the structured composition of EDD

Activity Model of Current Process

Specifies the flow of inputs/outputs and controls, including sequence for coordinating activities.

Partitions

Partitions (swimlanes) show responsibility for each activity.

Processing Server – 9 programsEngineer – 9 copies/moveUnix – 6 Perl scriptsReview Team – 6 QCsMac Client – 1 activity

Alternative (Attenex or Autonomy) process used to compare with the current process

Processing Server – 1 programEngineer – 2 copies/moveUnix – 2 Perl scriptsReview Team – 2 QCsMac Client – 1 activity

Advantages of Alternative Process

• Fewer manual steps

• Reduced probability of error

• Simpler to maintain

• Easier to train

• Less rigid process

• Shorter time to process documents

Agenda

•Approach - SysML Model

•Analysis - Trade Study

•Evaluation - SysML Usability

Net Present Value Probability Distribution

• The goal was to model the financial impact of each alternative over three years using Net Present Value (NPV).

• NPV is a capital budgeting technique used to estimate and compare cash flows for competing systems and projects.

• For each system the Net Cash Flow was decomposed, modeled, and run in a Monte Carlo simulation to generate NPV estimates.

• The results are NPV probability distributions for each alternative

t – time

n – total project time

r – discount rate

Ct – net cash flow

Co – Initial capital expenditures at time zero

Net Present Value

• Compared to the baseline, the alternative systems increase the processing speed and the ability to accept projects. The trade off is increased costs.

• Autonomy:

– $2,000,000 initial cost

– $250,000 annual maintenance cost

• Attenex:

– $500 per gigabyte processed operational cost

• How does the increased ability to accept new projects and the increased costs impact the profitability of the systems?

NPV – Results

Baseline Mean: $12.0 Million

Attenex Mean: $12.3 Million

Autonomy Mean: $16.2 Million

0

0.01

0.02

0.03

0.04

0.05

0.00 2.50 5.00 7.50 10.00 12.50 15.00 17.50 20.00 22.50 25.00 27.50

Revenue (Millions)

Baseline

Attenex

AutonomyProbability

NPV ($)

NPV Probability Density FunctionThe results assume that the alternatives will increase the number of projects allowed into the system by twice the baseline.

Autonomy is being pulled into a higher range of profitability!

Conclusions & Recommendations

• The model shows that by increasing the opportunity to accept new projects the alternative systems can overcome the increased costs!

• The future system for Huron will be a hybrid of the alternatives.• The process used for a particular project will be dependent on the clients’

requirements.

• The baseline system, while slower, provides a reliable and cost effective solution.

• For clients who choose higher speeds at higher costs Attenex would be an ideal fit. (Huron already owns licenses for the software!)

• It is critical to spread the costs of Autonomy across the three EDD groups. In effect distributing the responsibility for recouping the investment!

Agenda

•Approach - SysML Model

•Analysis - Trade Study

•Evaluation - SysML Usability

Purpose of the SysML Evaluation

• Aerospace asked us to evaluate SysML to determine how effectively SysML and Rational System Developer worked

• Evaluate SysML as a modeling language for designing systems• Evaluate SysML maturity• Determine how useful SysML is for systems engineering

design and evaluation• Evaluate IBM Rational System Developer

• Determine how well it supports SysML usage

Approach: Survey

• Created a Multi-Attribute Utility Assessment Evaluation Hierarchy survey• Survey contained 41 questions developed to assess the

strengths and weaknesses of SysML and Rational System Developer

• Questions were answered on a 1 to 5 Likert scale with 5 indicating a positive response

• Surveyed 8 OR680 Students using SysML• Electronic Data Discovery (EDD)• Tactical Surveillance Satellite (TSS)

Multi-Attribute Utility AssessmentEvaluation Hierarchy

Overall Utility

Effect On TaskPerformance

Usability

* Modified from Adelman & Riedel, Handbook for Evaluating Knowledge Based Systems, 1997

ProcessQuality

Product Quality General Ease

Of Use

Ease of Learning

Flexibility

Interface

Decomposed the evaluation criteria to evaluate differentaspects of SysML and RationalSystem Developer

Tactical Surveillance Satellite (TSS) Electronic Data Discovery (EDD)

• Values closer to 5 indicate positive responses

Utility Results

Averages TSS EDDOverall 4.1 4.2 4.0

Effect on Performance 3.6 3.7 3.5Process 3.6 3.8 3.6Product 3.5 3.6 3.4Usability 3.5 3.3 3.7Interface 3.4 3.3 3.5

Ease of Use 3.8 3.9 3.8Learnability 2.6 2.5 2.7Flexibility 3.7 3.4 4.0

Survey AnalysisSysML• Strengths

• Overall respondents felt SysML was a good language• Scored well in usability and flexibility

• Weaknesses• The main weakness in SysML is that it is difficult to learn

– Respondents took 20-40 hours to become a functional user

Rational System Developer• Strengths

• Rational System Developer scored highest in usability• Survey indicates that people found Rational System Developer fairly easy to use

• Weaknesses• Survey indicted low scores for ease of training• The Interface and product quality also scored lower than other areas

Recommendation to Aerospace• SysML

• SysML is difficult to learn and will require investment in training and time

– May not be practical for smaller systems or processes with limited complexity– However, if people are already trained, SysML diagrams ensure consistency

and provide effective communication across multiple disciplines

• Rational System Developer• Rational supported the creation of models and helped maintain

consistency• Process descriptions were created and analysis performed using Rational

and SysML

• SysML is well suited for complicated systems with significant hierarchical decomposition, systems common in the National Security Space domain

Summary

• Huron asked us to evaluate their current EDD system and two alternatives• Used SysML and NPV to perform the analysis• Determined that the best solution is a mix of the current

system for most clients and Autonomy for clients that require faster processing and can afford the increased cost

• Aerospace asked us to evaluate SysML to determine how effectively it can support system engineering design and analysis• Conducted a survey to help answer this question.• The survey found that SysML is a useful tool,

but the learning curve is steep

Acknowledgements

• Heather Howard, Shana Lloyd, and Julie Street, Aerospace Corporation

• Chris Genter, Huron Consulting Group• Professor Laskey, George Mason University• Sanford Friedenthal, Lockheed Martin• Professor Adelman, George Mason University• The TSS Team

• David Alexander, Kevin Sadeghian, Siroos Sekhavat, and Tom Saltysiak

Future Work

• Optimize Parametric Diagram to make the model executable

• Run executable model

• Compare executable model results with results obtained from Microsoft Excel

• Distribute SysML survey to future students for a larger sample and further analysis

Questions?

Backup

Component Diagram

Decompositions

Parametric Diagram

• Parametric Diagrams were created to express constraints between value properties and allow to perform an executable model.

• Executable model used to provide analysis for performance, safety, reliability, throughput, weight, cost, etc.• High Learning Curve• Lack of Time (Estimation of >20+ additional hours to learn SysML

limitations)• Inexperience with Simulation Toolkit (Estimation of >30+ hours to

execute with toolkit)• Inexperienced team with Java (Estimation of >70+ hours to learn

Java)

• Questions focus on either SysML as a language or IBM Rational System Developer as a tool

• Most questions will be rated on a scale of 1 to 5• Responses will be averaged together to determine a score

for each category• Sample Questions

• Overall, SysML improves the system design process.• Rational System Developer provides feedback when

processing user commands.• SysML was easy to learn.• I can easily add model elements to the System model.

Sample Survey Questions

Survey Example

10. Modeling a system with SysML is faster than the current process (i.e., Power Point)Strongly Disagree Strongly Agree

1 2 3 4 5

11. The SysML diagrams available were adequate for my project.Strongly Disagree Strongly Agree

1 2 3 4 5

Survey will have participant answer a series of questions

Webpage

mason.gmu.edu/~cchang7

General Status

ScheduleID Task Name Duration Start Finish

1 EDD Project Plan 67 days? Thu 2/1/07 Thu 5/3/07

2 Prepare Proposal 11 days Thu 2/1/07 Thu 2/15/07

3 Develop Project Scope 11 days Thu 2/1/07 Thu 2/15/07

4 Identify Models 6 days Thu 2/8/07 Thu 2/15/07

5 Background Research and Learn SysML 17 days? Thu 2/8/07 Fri 3/2/07

6 Detail SysML Metrics & Measures 7 days Wed 2/14/07 Thu 2/22/07

7 Detail Huron Metrics & Performance 7 days Wed 2/14/07 Thu 2/22/07

8 Contact Aerospace 1 day? Thu 2/22/07 Thu 2/22/07

9 Determine a day to visit Huron 7 days Wed 2/14/07 Thu 2/22/07

10 Research Background Literature 11 days Thu 2/8/07 Thu 2/22/07

11 Research Autonomy 11 days Fri 2/16/07 Fri 3/2/07

12 Research Attenex 11 days Fri 2/16/07 Fri 3/2/07

23 Model Processes in SysML 42 days Fri 2/16/07 Fri 4/13/07

24 Implement Model of Current Process in SysML 21 days Fri 2/16/07 Thu 3/15/07

25 Implement Attenex Model in SysML 21 days Fri 3/2/07 Thu 3/29/07

26 Implement Autonomy Model in SysML 21 days Sat 3/3/07 Fri 3/30/07

27 Optimize Models and Perform Trade Analysis 21 days Fri 3/16/07 Fri 4/13/07

28 Track SysML Issues and Metrics 56 days Fri 2/9/07 Thu 4/26/07

29 Learn SysML 16 days Fri 2/9/07 Fri 3/2/07

30 Use SysML 51 days Fri 2/16/07 Thu 4/26/07

31 SysML Training 0 days Sat 3/3/07 Sat 3/3/07

13 Develop Final Presentation 17 days Wed 4/4/07 Thu 4/26/07

14 In Class Deliverables 56 days Thu 2/15/07 Thu 5/3/07

15 Proposal and Presentation 0 days Thu 2/15/07 Thu 2/15/07

16 Status Report 0 days Thu 2/22/07 Thu 2/22/07

17 Progress Report 0 days Thu 3/8/07 Thu 3/8/07

18 Status Report 0 days Thu 3/22/07 Thu 3/22/07

19 Formal Progress Presentation 0 days Thu 4/5/07 Thu 4/5/07

20 Dry Run of final Presentation 0 days Thu 4/26/07 Thu 4/26/07

21 Final Project Presenation 0 days Tue 5/1/07 Tue 5/1/07

22 Final Report 0 days Thu 5/3/07 Thu 5/3/07

3/3

2/15

2/22

3/8

3/22

4/5

4/26

5/1

5/3

1/28 2/4 2/11 2/18 2/25 3/4 3/11 3/18 3/25 4/1 4/8 4/15 4/22 4/29January February March April

NPV Backup

NPV - Formula

Where:

t – time

n – total project time

r – discount rate

Ct – net cash flow

Co – Initial capital expenditures at time zero

NPV - AssumptionsNumber of Projects Limitations: The number of projects entering into the system can not be greater than the

maximum level of availability.

Projects Start and Completion Time: All projects started in a month are assumed to be completed within that month. In practice this assumption can be interpreted as larger scale projects are started early in the month while smaller projects are started later in the month.

Minimum Revenue: $2500 is the minimum amount of revenue accepted for a job.

Autonomy Costs: The Autonomy system has an initial cost of $2 million dollars and an operational cost of $250,000 annually.

Attenex Costs: The Attenex system has an operational cost of $500 dollars per GB processed.

Prospective Projects: The level at which prospective projects are found is consistent for all systems.

Availability Parameter: The availability parameter is being used to model the size and availability of the queue for incoming projects.

Pricing Scheme: The pricing scheme is constant for each system over the three year period. No adjustments have been made to the pricing schemes of the higher cost alternatives.

Migration Costs: With the exception of initial software costs, all migration costs are ignored in this model.

NPV- Revenue Inputs (1)

Annual Revenue: The annual revenue is the sum of twelve monthly revenue estimates.

Monthly Revenue: The monthly revenue is the sum of the revenue for each job accepted and completed in a month.

Revenue per Project: The revenue per project is the amount of revenue in dollars that a generated by a project.

Projects Accepted: This value is the total number of projects entered into the system each month.

NPV- Revenue Inputs (2)

Maximum level of System Availability: The maximum level of system availability is the largest number of projects that can enter into the system each month.

Number of Prospective Projects: The number of prospective projects describes the number of projects that are available to be entered into the system.

Number of Staff: The number of staff plays a critical role in limiting the number of jobs that can be entered into the system each month.

Processing Speed: Processing speed describes the rate at which projects can be pulled through the system.

NPV- Cost Inputs

Initial Costs: The costs used to procure new software and equipment for the alternative systems at the onset of the migration. The initial costs are incurred once at the beginning of the project.

Maintenance Costs: Monthly costs associated with maintaining the software and hardware systems. The maintenance costs include repairing machines, software upkeep and spare parts.

Salary Costs: Monthly costs related to employee salaries.

Operational: Monthly costs related to procuring additional equipment, software and the overhead costs related to the building and facilities.

NPV – Parametric Diagram

Staff

Monthly RevenueProjects Accepted × Revenue per Project

Maximum System Availability

Revenue per Job

Projects Accepted

Potential Projects

Annual RevenueSum of Monthly

Revenues

Annual Costs

Sum of Monthly Costs

Monthly CostsOperations + Salaries + Maintenance

Operations Maintenance Salary

NPVAnnual Revenue – Annual Costs

Processing Speed