Design of a Procedure Analysis Tool (PAT) for Affordable ...Design of a PAT for Affordable Human Factors Certification of Aviation Devices D ESIGN OF A P ROCEDURE A NALYSIS T OOL (PAT)

Mr. George Blaha Department of Systems Engineering and Operations Research

Design of a PAT for Affordable Human Factors Certification of Aviation Devices

DESIGN OF A PROCEDURE ANALYSIS TOOL (PAT)

FOR AFFORDABLE HUMAN FACTORS

CERTIFICATION OF AVIATION DEVICES

Team Members: Heather Johnson, Houda Kerkoub, Shaida Latifi, Mumtahina Mahmud, Alexis Nittoli

New Regulation

More thorough device

certification

DER



Agenda

Context

Stakeholders Analysis

Problem Statement

Proposed Solution

Objectives Hierarchy and Requirements

Method of Analysis

Prototype: Procedure Analysis Tool (PAT)

Business Case - Alternatives

2



Flightcrew Errors as Major Accidents Factor

139 commercial jet aircraft accidents between 1993-2002 67% of error was due to flight crew

In 2009, flight crew errors were 56%

3

Source: Statistic analysis of airplane accidents, Copyright 2005-2014



In Response to Accidents

Issuance of FAR 25.1302 enforcing Human Factors

evaluation in the device

certification process.

4

Complemented by

AC 25.1302

“the applicant1 must show that […]

flightcrew […] can safely perform all

of the procedures associated with the

systems' and equipment's intended

functions”

“Section 25.1302(b)(2) requires controls and

information be accessible and usable by the

flightcrew in a manner consistent with the

urgency, frequency, and duration of their

tasks”

Requirements



Standard Operating Procedures (SOP): Instructions

with purpose of documenting how to perform a

routine or emergency activity and to ensure safety,

consistency, and quality of a product.

SOP can be found in:

Airline Operations Manual (part B)

Training Manuals

Airline Computer Based Training (CBT)

Test Case: Swiss European RJ100 (CBT)

Airline Standard Operating Procedures (SOP) and Tasks

Item Count

Procedures 104

Tasks 642

Sub-tasks 1,263

5



Device

Example device is the Multifunction Control Display Unit

(MCDU) incorporates:

A small screen

A keyboard or touchscreen

Main input to the flight management computer (automation)

Swiss European CBT displays 62 tasks

6



Selected Tasks on Device 7



Time on Task 8

NORMINV(RAND(), 2,

1)

LOGINV(RAND(), )

EXPON.DIST(RAND(), 1,1)

Replication

500

𝑖=1



Tasks and Operator Actions

Example Procedure: Change Hold Parameters Stimulus: ATC call: Swiss XX Hold at Present Position, 1 nm legs, left turns, ECHT 1220Z Note: this task is a second task after completing the Hold at Present Position Task, hence the Working Memory retrieval as a trigger to the task Identification

9

# Operator Action (OA) Operator Action Category (OAC) Rand Num

1 from clearance to hold, remember the hold parameters working memory item retrieval 1.104217234

2 Identify task "change the directiion of turn, inbound course, and speed" decision (TSL Id) 0.55805178

3 select functions TURN DIR, INBD CRS, and TGT SPD TSL Select 0.103369032

4 L2: turn dirction is right visual cue: Rare frequency 1.65063597

5 LM: remember TURN DIR means direction of the turn long term memory item retrieval 4.250202572

6 LM: remember you can change the direction by a direct scratchpad entry long term memory item retrieval 4.250202572

7 LM: remember how to use the scratchpad: make the entry then select the field long term memory item retrieval 4.250202572

8 letter L on the alphanumerical keys visual cue: Rare frequency 1.65063597



Task Decomposed into 3 Operator Actions (OA) 10

NORMINV(RAND(), 2,

1)

LOGINV(RAND(), )


Replication

500

𝑖=1



Probability of Failure to Complete Procedure 11

L. Sherry and e. al., "Estimating the Benefits of Human Factors Engineering in NextGen Development: Towards a Formal

Definition of Pilot Proficiency," in 9th AIAA Aviation Technology, Integration, And Operations Conference (ATIO), 2009.



Probability of Failure to Complete

+ + +

Working memory retrieval Visual Cue TSL Select

TSL ID

Simulating flightcrew

performance and

keeping track of their

timing while executing

procedures on a device

enables affordable

automation for

task/usability analysis.

12



Regulation Guidelines (Advisory Circular)

Source: AC 25.1302

2 Methods for Means of

Compliance:

13

Advisory Circular 25.1302 provides

guidelines for Methods of Compliance with

FAR 25.1302

Focusing on requirement b-2

Implementation of accessible and

usable control by flight crew

Urgency

Frequency

Task duration

Inspection testing

Human in the loop testing



The Current way of Certification (Pre-Regulation)

Human in the Loop Inspection

4 hours per crew

per 30 functions

crew

14

Tactile evaluation

Color differences

Position of device and reachability

Readability in both sunlight and darkness

Captain and first officer (2 crew members) are

tested within in a simulator or flight test aircraft

This test must be completed 10-20 times to

guarantee statistical significance



The Problem: Certification as Required by FAR25.1302

For 600 functions device, increase

above time 20 folds

crew

New Regulation

More thorough

device certification

Inspection Human in the Loop

15

4 hours per crew

per 30 functions



Context


Problem Statement

Proposed Solution


Method of Analysis



Agenda 16



Stakeholders

Federal Aviation

Administration

Passenger

Aircraft

Human

Factor

Specialist

(Inspector) Designated

Engineer

Representative

(DER)

Provide

Airworthiness

Certification

Approve

Technical Data

Support the

Certification

Certify HF

Device

Certify HF

Device

Build

Demand

Increase

Safety, Lower

Cost of Travel

Push to Sell

Device

Congress

Change

Regulations

Air Traffic

Controller

Communicate Air

Traffic Information

Tension

Interactions

Primary

Secondary

Tertiary

Employs

Pilots

Aviation

Manufacturer

Pilot

Aircraft

Operator

Product

Manager

17

The AC25.1302 describes Evaluation as

“assessments of the design conducted

by the applicant (or their agent), who

then provides a report of the results to

the FAA”

Maintain Safety

through

Certification



Designated Engineering

Representative

DERs

FAA Human Factors Specialist Aviation Manufacturer

Product Manager

DER must perform

certification tests

and report to FAA

inspector while

being employed by

applicant for

certification

Stakeholder Tensions upon DER 18



Agenda

Context


Problem Statement

Proposed Solution


Method of Analysis



19



Costs of Tasks Pre and Post FAR 25.1302

Inspection Costs $10,000 100 hours

Human in the Loop Costs $979,000 1600 hours

Total $989,000 1700 hours

POST FAR 25.1302: Evaluation of 600 tasks MUST be completed

20

Inspection Costs $500 5 hours

Human in the Loop Costs $14,000 80 hours

Total $14,500 85 hours

PRE FAR 25.1302: Evaluation of 30 over 600 tasks ONLY

Certification time:

TProcess (x) = TINSP(x) + THITL(x) = 5x

CProcess (x) = CINSP(x) + CHITL(x)= 1600x



Problem Statement 21

Time and Cost increase dramatically due to

FAR 25.1302

Pre – FAR 25.1302 Post – FAR 25.1302

$14,500 $989,000

85 Hours 1700 Hours

The new regulation FAR25.1302 requirement to evaluate all

Procedures is cost and time prohibitive if it has to be accomplished

by the currently approved Human-in-the-Loop testing.



Additional Problems

1. Stakeholders tension creates pressure on the DER* who is

both employed by the applicant and must report to the

FAA inspector during the certification process

2. Current means of compliance are subjective

3. Lack of cockpit interaction in the current certification

process

22

*DER: Designated Engineering Representative



Context


Problem Statement

Proposed Solution


Method of Analysis



Agenda 23



Concept of Operations

The PAT simulates crew performance and

tracks their timing while executing

procedures on a device enabling affordable

automation for task/usability analysis.

Tool for DER that substitutes the current

certification process by PAT testing on

ALL tasks.

Dedicate HitL testing to only procedures

with high PFtoC.

24

DER



Current Process vs. Solution

SOLUTION- New Process with PAT

TAlt2,2 = TINSP(x) + TPAT +10%THITL(x) = 2x

CAlt2.2 (x) = CINSP + CPAT + 10%CHITL = 500x

25

Note: Calculations based on inspector wage, pilot wage, and inspection time

Current Process TProcess (x) = TINSP(x) + THITL(x) = 5x

CProcess (x) = CINSP(x) + CHITL(x)= 1600x



Proposed Solution

Provide automated evaluation which is significantly faster and

cheaper

Eliminate stakeholders’ tensions by providing rapid, yet

objective means of analysis

Enable implementation of accessible and usable control and

information consistent with (1) urgency, (2) frequency, and (3)

task duration

26



Context


Problem Statement

Proposed Solution


Method of Analysis



Agenda 27



Mission Requirements

MR.1 PAT shall enable compliance with the FAR 25.1302

requirements

MR.2 PAT shall reduce the post FAR25.1302 certification

cost by at least 65% to match the pre-FAR 25.1302

cost

MR.3 PAT shall remove the stakeholder tensions between

the FAA, the Applicant, and the DER

28



Technology Requirements 29

TR1. PAT shall use a Graphical User Interface (GUI)

to enable user input and tool output

display

TR2. PAT shall use a Database to store Operator

Performance data

TR3. PAT shall accept up to 600 procedures for

decomposition

TR4. PAT shall use a lookup function to map the

OA to their statistical distributions

TR5. PAT shall use an algorithm to generate random

numbers and sum them into total

procedure time

TR6. PAT shall use Monte Carlo Simulation to

simulate 500 operators performing the SOP

procedure

DER



Objectives Hierarchy for PAT 30

O.1 Maximize Compliance with FAR 25.1302 (w=58%)

O.2 Minimize Operational Costs (w=16%)

O.2.1 Minimize Set Up Costs

O.2.2 Minimize evaluation cost

O.3 Optimize Performance (w=26%)

O.3.1 Maximize Availability of PAT tool

O.3.2 Maximize Maintainability

O.3.3 Maximize Reliability of Procedure Analysis Tool



Context


Problem Statement

Proposed Solution


Method Of Analysis



Agenda 31



Process Modeled 32

OA: operator action

OAC: operator action categories

SOP: standard operating procedure

PFtoC: probability of failure to complete



IDEF0 Diagram – System Steps

DER decomposes the procedures

into OA and categorizes them

OAC

PAT maps the OA to its statistical

distribution

PAT sums up all the OA times

PAT performs Monte Carlo

simulation

PAT displays output

DER inputs threshold

DER inputs acceptable PFtoC

PAT displays results

PAT provides option to perform

sensitivity analysis

DER selects option to perform

SA

DER inputs SA parameters

PAT shows results

33



Human Factors OAC Database 34

Literature Review

Experiment

Desing not started

Operator Action Category (OAC) Distribution

button push Normal(0.3, 0.01)

decision (TSL Id) Normal(0.5,0.002)

decision/choice Lognormal(2,1)

flight controls manipulation (feet) Normal(2.31, 0.651)

flight controls manipulation (hands) Lognormal(1.12, 0.231)

listen to audio (ATC Clearance, Aural Warning) exponential (1)

long term memory item retrieval Lognormal(1.21, 2.1)

MCC callout Normal(2,1)

MCC readback Normal(1.5, 0.2)

Talk to ATC (including Readback) exponential (0.9)

thrust levers manipulation Triangular(0.1,2,3.5)

TSL Select Normal(0.1,0.002)

visual check Normal (1, 0.5)

visual cue: Routine Normal( 0.632, 0.011 )

visual cue: Moderate frequency Normal( 1, 0.02 )

visual cue: Rare frequency Normal( 1.632, 0.03 )

working memory item retrieval Lognormal(0.737, 1.21)



Operator Actions Distribution 35

Working Memory Log (0.737, 1.21)

Participant 1: Time keeper From the Standard Operating Procedures dictate a task. Ex:

Press HOLD

Press 6R

Review the default

Press EXEC to activate holding patterns

After a 51 second interval

Ask designated questions

Repeat it for 10 times with 5 subjects 50 data points

Tabulate the data in Microsoft Excel

Participant 2: Subject Follow Participant 1’s instructions

Fre

quen

cy

Long term memory Expo(1.21,2.1)

Completion time (sec)

Fre

quen

cy

Same experiment was performed

after 10 days but without showing

the task steps to subjects.

Completion time (sec)



Sequential Variables Process

Replication

36

NORMINV(RAND(), 2,

1)

LOGINV(RAND(), )


500

𝑖=1

Assumptions:

Independent Variables

No Dual Tasks Performance



Context


Problem Statement

Proposed Solution


Method of Analysis



Agenda 37



Example Task Decomposition – “Initialize Position” 38



Prototype PAT

1. DER enters

Decomposition here

2. DER categorizes the

OA here 3. PAT generates

random numbers here

and computes sum on

the last row

4. Pressing this

button displays

the 500 task

times here

39

5. Pressing this button

displays frequency

distribution



Results - 15 Tasks 40

Acceptable PFtoC the task was set to 2.5%

*PFtoC: Probability of Failure to Complete the task

# ProcedurePAT Entry

time [hr]

PAT Run

Time [sec]Mean Std

Threshold

= 3std

from

mean

*PFtoC

Percent

OAC -

Visual Cue

(%)

1 Initialize the Position 1:05 3.76 238 192 815 1.60% 47.27%

2 Enter Company Flightplan 1:30 3.21 219 175 742 2.60% 40.82%

4 Check Flgihtplan Progress 1:00 3.66 217 328 1200 1.00% 45.16%

5 Switch MCDU ON 0:15 19 92 98 386 3.40% 41.18%

6 Enter Runway and SID info 0:30 16 145 145 578 1.00% 59.57%

7 Enter flightplan manually 1:28 13 208 175 733 2.60% 52.00%

8 Hold at Present Postion 2:30 10 238 192 815 1.50% 39.00%

9 Change turn direction, speed… 0:21 14 135 133 533 1.60% 39.29%

10 Check Climb/Cruise Settings … 0:23 13 107 104 419 1.80% 48.00%

11 Change Speed and Altitude … 0:36 13 89 62 275 2.20% 51.51%

12 Select Arrival and Approach… 0:47 14 170 123 539 2.20% 61.70%

13 Enter a Runway Extension 0:36 13 90 99 385 2.40% 41.18%

14 Enter Hold ExitTime 0:26 13 215 165 708 2.40% 50.00%

15 Select Speed Restriction 0:40 11 105 145 540 0.4% 48.0%

-- Total Time 12:47 173.44 -- --

-- Average 00:47 11.5626667 2% 47%



Preliminary verification and validation results using GMU

students performing tasks on a device.

Verification and Validation 41

Normal Distribution with Mean of

1.06 and Standard Deviation of 0.43 Lognormal Distribution with Mean of

2.93 and Standard Deviation of 1.04

Experiment with Cognition Experiment without Cognition



Context


Problem Statement

Proposed Solution


Method of Analysis



Agenda 42



1. A La Carte Model: $200 per function

2. License Model: $75,000 per year with recurring annual fee of

$25,000 thereafter

3. Application Purchase Model: $500,000

Business Models used for PAT 43



Market Analysis 44

Market prospection:



Acquisition of Customers

Year 1 1 new “A La Carte” customer

Year 2 2 new “A La Carte” customers

1 customer shift to “License” Model

Year 3 1 new “A La Carte” customer

2 customers shift to “License” Model

Year 4 7 new customers use “License” Model

Year 5 1 customer shifts to “Purchase” Model

Phase 1: First 5 years

Phase 2: Following 5 years – Radom numbers generated per following uniformly distributed

intervals

A la Carte

New: between(0,3)

Shift to License: between(1,2)

*Lost customer: between(0,1)

License Purchase

New between(0,7)

Shift to Application: between(0,2)

*Lost customer: between(0,2)

45

Application Purchase

New: between(0,1)

*Lost: customer 0

* Lost customers modeled over the 10 years period



Market Penetration per Business Model 46



Startup Costs – Software development: $250,000

– Marketing, sales, business management: $250,000

Recurrent Costs: – $250,000 scales with consulting business and product

support services growth

Annual Costs 47

year0 -$ 500,000$ (500,000)$ -200%

year1 50,000$ 750,000$ (700,000)$ -193%

year2 350,000$ 1,000,000$ (650,000)$ -165%

year3 875,000$ 1,250,000$ (375,000)$ -130%

year4 2,550,000$ 1,500,000$ 1,050,000$ -30%

year5 3,850,000$ 1,750,000$ 2,100,000$ 20%

Time ROIProfitCumm. CostCumm. Revenue



Return on Investment and Breakeven Point

At end of year 5, ROI is 20%

Breakeven point is forecasted to occur after year 4:

48

ROI -193% -165% -130% -30% 20%



Risk Analysis

Risk Area Failure Severity (S) likelihood (L) Detectability (D) RPN

market size is too small 8 1 5 40

market is not growing 6 1 5 30

regulatory trends are not aligned with service/product 9 5 6 270

not able to deliver service in time 5 3 3 45

staff issues 5 5 5 125

fall under worse scenario or below (breakeven and ROI) 10 4 1 40

recession 7 5 6 210

business

uncertainty

operational risks

financial model

uncertainties

FMEA Matrix (1 to 10 scale)

49



The PAT reduces the evaluation time by 70% per function and therefore solves the problem imposed by FAR25.1302.

The PAT was demonstrated to work on 15 Procedures on the MCDU device.

Evaluating the 15 procedures with PAT took 12:47 hr. entry time, 11.56 sec. average runtime, and analysis showed that PFtoC ranged between 0.4% and 2.6%. Also, 3 over 15 procedures (20%) were flagged for HitL testing.

The Visual Cue category gathered the greatest OAC percentage which highlights the importance of semantic cues to recognizing emerging mission situations and prompting the next step of the procedure process.

Further exploration of the Operator Action Categories and their impact on the PFtoC is recommended.

Conclusions and Recommendations 50



Thank You



Questions?



Task Decomposition follows TSL 53

• TSL framework showing

interaction between all three

agents (lifelines)

• Arrows show flow of

information between agents

through six steps

• Cognitive actions are

shown as recursive arrows

on the Operator

• TSL Steps are shown in red

• Top to bottom shows time

progression

Documents

Design of a Procedure Analysis Tool (PAT) for Affordable ...Design of a PAT for Affordable Human Factors Certification of Aviation Devices D ESIGN OF A P ROCEDURE A NALYSIS T OOL (PAT)