Army Materiel Systems Analysis Activity (AMSAA) …...Reliability Scorecard PSM Workshop 21 March 2018 Army Materiel Systems Analysis Activity (AMSAA) Resources 4 • Provides standard

Army Materiel Systems Analysis Activity (AMSAA) Resources

AMSAA

Dr. David Mortin

21 March 2018

PSM Workshop 21 March 2018 1Army Materiel Systems Analysis Activity (AMSAA)

Resources

DISTRIBUTION STATEMENT A - Approved for public release.

Distribution is unlimited

Resources

• Design Modeling

• Assessment tools• Scorecard• Reliability Growth Models

• Data• Sample Data Collection• CBM

• Metrics

• Analysis

PSM Workshop 21 March 2018Army Materiel Systems Analysis Activity (AMSAA)

Resources2

https://www.amsaa.army.mil/CRG_Tools.html

Preparing for a Challenging Environment


Resources3

Reliability Scorecard


Resources4

• Provides standard method to identify reliability risk areas

• Based on IEEE, academia, Raytheon, Alion, RIAC, GEIA recommendations, best practices, etc.

• Structured approach allows refinement and improvement.

• General Scorecard has 8 Categories (40 elements)

• Software systems version has 7 Categories (57 elements)

Reliability Requirements and

PlanningTraining and Development Reliability Analysis Reliability Testing

Reliability activities that are integral

to design and testing are clearly

identified and incorporated into the

program Integrated Master

Schedule

Sufficiently-sized reliability

engineering staff is directly tied to

the design team

Conduct failure modes, effects,

and criticality analysis (FMECA)

and fault tree analysis (FTA);

crosswalk to low-level testing and a

failure mechanism analysis to

ensure programmatic coverage

Conduct low-level testing early to

identify and mitigate failure modes.

Conduct highly accelerated life

testing (HALT) and highly

accelerated stress screening

(HASS)

Supply Chain Management Failure Tracking and Reporting Validation and Verification Reliability Improvements

Plan and establish mature and well-

documented manufacturing procedures

Utilize failure reporting, analysis and

corrective action system (FRACAS)

Modify testing and procedures based

on failure occurrencesDocument lessons learned

Sample Elements


Resources5

Reliability Design Analyses

This Circuit Board Fails Due

to VibrationThis Circuit Board Does Not

(just by adding two screws)

Initial Design

Electronics in an Army Hand Held Device

Design after Physics-of-

Failure Vibrations Analysis

Design modification costing almost nothing changes product from being unreliable to reliable

Thermal Analysis

Provided

Additional Insights


Resources6

System-Level Modeling

Full-system Modeling provides tremendous insights that reduce testing and

eliminate failures (pre and post fielding)

• Mitigate sources of failure

• Quickly develop solutions for failures

• Positively impact

design


Resources7

Root-Cause Analysis

Critical Components:

• Stresses, forces,

accelerations, etc. are

available for

components in the

model

• Models allow for quick

visualization of

multiple fixes or design

enhancements

• Identify weak spots and redesign in

virtual environment – not test-fix-

test (Too Expensive!)


Resources8

Schofield Barracks

Joint Base Lewis-McChord

Ft. Irwin

Ft. Carson

Ft. HoodFt. Bliss

Ft. Rucker

Ft. Campbell

Ft. Stewart

Vilseck

Kuwait

Poland

Ft. Riley

ANAD

• 197 contractors at 11 CONUS and 3 OCONUS locations

• Over 7,044 ground systems and 508 aviation systems

• Reliability Metrics

• Maintenance Hours

• Trends

• New Maintenance Pilots

• Fixing Specific Field Issues

Sample Data Collection & Analysis


Resources9

Download to Laptop

(Includes at platform Diagnostic

Trouble Codes)

Analysis

Algorithms

Digital Source Collector (DSC)

& JPRO

Server

AMSAA SDC&A

Complete & Accurate

Field Level Maintenance

Data

CBM+: Fusing Data to Provide

Actionable Information

Web-based CBM+ Tools

Motor Pool Maintenance

Management

Fleet Management

Vehicle Health Alerts

Using Data

Analytics of 200+

Billion Sensor

Readings &

Maintenance

Events to Provide

Actionable

Information at

Field and Fleet

Levels

AMSAA has 2,300+ DSCs

installed on vehicles at 8

world-wide locations


Resources10

Brakes ParametersFront Axle Speed

Relative Speed, Front Axle, Left Wheel

Relative Speed, Front Axle, Right Wheel

Vehicle Speed

Brake Switch

Engine ParametersAccelerator Pedal Position

Barometric Pressure

Boost Pressure

Coolant Level

Engine Coolant Temperature

Engine Load

Engine Oil Pressure

Engine Oil Temperature

Engine Speed

Fuel Rate

Fuel Temperature

Injection Control Pressure

Intake Manifold Temperature

Engine Percent Torque

Vehicle Speed

PTO Mode

Transmission ParametersTransmission Oil Temp

Transmission Range Attained

Transmission Range Selected

Transmission Output Shaft Speed

Input Shaft Speed

Transmission Actual Gear Ratio

Transmission Current Gear

Transmission Selected Gear

Torque Converter Lockup Engaged

Diagnostic Trouble CodesEngine

Transmission

Brakes

Typical CBM+ Data Collected

CBM+ to Enable

Data Driven Fleet

Management

Improved equipment

Readiness

Cost savings

Reduced repair time

Value added

maintenance activities

Correct parts ordered

Better Options for Reliability Prediction


Resources11

0 500 1000 1500 2000 2500 3000 3500 4000 4500

System 14

System 13

System 12

System 11

System 10

System 9

System 8

System 7

System 6

System 5

System 4

System 3

System 2

System 1

Mean Time Between Failure (Hours)

Demonstrated

Prediction 1) Hard to keep handbooks up-to-date with current technology

2) Represent only a small portion of the overall system failure rate

3) Does not consider critical factors that influence reliability (e.g. thermal cycling)

4) Does not provide insight on how or why the system will fail

5) Data not current (lagging years)

Economic Useful Life Analysis


Resources12

0

2

4

6

8

10

12

14

16

18

20

No

n-M

iss

ion

Ca

pa

ble

Ac

tio

ns

(p

er

XX

XX

mil

es

)

Odometer Reading (miles)

Based on 6 years of Sample Data Collection field data

Vehicles range in age from 3 to 12 years old

Represents the 10th, 50th, and 90thpercentiles

for each mileage bin

For this vehicle, no statistical indications of aging over the period

Multiple ways for PSMs to analyze to ensure optimum Economic Useful Life decisions


Resources13

Leveraging Data and Information to the Maximum Extent Possible

σf Subsystem Strength

Probability

of

Failure

Using Physics-Based-

Reliability methodology

to determine impacts of

weight or mission

changes for ground

systems

Field data from

existing systems

combined with

engineering modeling


Resources14

14

Reliability Metric Differences Pre and Post Fielding

Environment Incident Classification Metric

Test Test incidents classified using FD/SC (i.e.

EMA, UMA, OMF, SA, EFF)

MTBSA,

MTBEFF

Field Unscheduled maintenance actions classified

using TM-10 PMCS (i.e. NMC)

MTBNMCA,

MTBUMA

How can we compare

test metrics to field

metrics?


Resources15

A Case Study – Ground Vehicle

Test Field

Available Data Failure data collected via TIRs

“Detailed” description of

maintenance performed

Unscheduled maintenance and

associated part replacements

data collected in field

“Basic” fault description,

correction narrative and failure

description (if applicable or

known)

Mapped NSN’s that cause NMC

events in the field to test incident

reports

Common rule sets

in the field were

applied to test data


Resources16

Ground System Reliability Metrics

Adjustments to test data allow for a more direct comparison of test and field

reliability metrics.

Statistical tests can be utilized to determine if there are any significant

differences in the test and field data sets.

Mean Miles

Between

NMC Action

(MMBNMCA)

Total Mileage

# NMC Actions=

Reliability Comparison

*Also contrasted to baseline

requirement

Another way to

compare pre and post

fielding reliability is to

score the field data as

was done in testing


Resources17

80

114

175

215220

CAP1

CAP2

CAP3

CAP4

0

50

100

150

200

250

300

Idealized Curve Customer Test Initial DT LUT LUT Excursion IOT

$869 M$894 M

$1,103 M

$1,701 M

$2,457 M

Test Time (Hours)CAP – Corrective Action Period

Reliability Growth and its Impact On Support Costs

Me

an

Tim

e B

etw

ee

n F

ail

ure

Reliability Growth Planning,

Tracking, & Projection tools

along with the reliability

scorecards are available for

free to US Government

organizations and supporting

contractors

https://www.amsaa.army.mil/CRG_Tools.html


Resources18

Establishing Solid Reliability Growth Strategies

Category Low Risk Medium Risk High Risk

< 70% 70 - 80% > 80%

IOT&E Producer’s Risk ≤ 20% 20+ - 30% > 30%

IOT&E Consumer’s Risk ≤ 20% 20+ - 30% > 30%

Management Strategy < 90% 90 - 96% > 96%

Fix Effectiveness Factor ≤ 70% 70+ - 80% > 80%

< 2 2 - 3 > 3

Time to Incorporate and

Validate Fixes in IOT&E

Units Prior to Test

Adequate time and

resources to have fixes

implemented & verified

with testing or strong

engineering analysis

Time and resources for

almost all fixes to be

implemented & most

verified w/ testing or

strong engineering

analysis

Many fixes not in place by

IOT&E and limited fix

verification

Initial MTBF

(DT) Goal MTBF

Potential Growth MTBF

(DT) Goal MTBF

Risk

assessment of

curves now

included in

some

reliability

growth

modeling tools


Resources19

Supply Chain & Inventory Analysis Demand Planning

Analysis of Requirements,

New Logistics Concepts,

and PBL Arrangements

Army Prepositioned Stock Computations

$- $1 $2 $3 $4 $5 $6 $7 $8 $9

$10

0-2Years

2-5Years

5-10Years

10-20Years

20-50Years

50+Years

NFD

Bill

ions

Insurance Items BDROSafety Level LeadTime Rqmt24 Months Dmd Econ/LOT RetnLOT-Excess Excess

Establish business based

performance metrics to track

efficiency and value added

Assess requirements feasibility

along with mission impact

Selected Essential Stocks for Availability Methodology (SESAME)

Optimum stock levels and identification of excess inventory Analyzing accuracy of forecasts

Impact of multi-level additive

manufacturing and its potential

effects on demand planning

Optimum Stock Requirements Analysis Program

Optimized packages for Class I, II, IIIP, IV and IX

Stocks, Forecasting, and Requirements Assessment


Resources20

Materiel Management

Supply Chain

Inventory Optimization

Demand Planning

Army Prepositioned Stocks

Field Operations

Sample Data Collection & Analysis

Operational Sustainment

Condition Based Maintenance

Reliability Centered Maintenance

Acquisition Support

Logistics Footprint

Level of Repair

Life Cycle Cost

Schedule Risk

Reliability

Center for Reliability Growth

Physics of Failure

Requirements Determination

Test Support & Efficiencies

Operational Energy

Fuel Consumption

Power Sufficiency

Expeditionary Basing Energy

Operational Logistics Planning

75%

80%

85%

90%

95%

100%

$0.0 $0.5 $1.0 $1.5 $2.0

SA

$ Safety Level (Billions)

99% SA

85% SA

90% SA

95% SA

Resources and Contact Info

David Mortin, Ph.D.

410.278.6248

[email protected]

Documents

Army Materiel Systems Analysis Activity (AMSAA) …...Reliability Scorecard PSM Workshop 21 March 2018 Army Materiel Systems Analysis Activity (AMSAA) Resources 4 • Provides standard