Division of Information Management Engineering User Interface Laboratory 11 Fall 09 Human Interface UI Evaluating Design Proposals for Complex Systems

Division of Information Management Engineering

User Interface Labora-

tory

11

Fall 09’ Human Interface

UI 연구실백 지 승

Evaluating Design Proposals for Complex Systems with Work Domain Analysis

Neelm NaikarDefence Science and Technology Organization, Melbourne, Austrailia

Penelope M. SndersonSwinburne University of Techonology,

Melbourne, Austrailia

1



tory

11

Fall 09’ Human Interface Division of Information Management Engineering


tory

#11

Contents INTRODUCTIONEvaluating Designs for Complex SystemsStandard Technique for Evaluating DesignsA Work Domain Analysis-Based Approach for Evaluating Designs

EVALUATING DESIGNS FOR AIRBORNE EARLY WARN-ING

AND CONTROLAEW&C Work Domain AnalysisUsing Work Domain Analysis to Evaluate AEW&C DesignsBenefits and Challenges of Using Work Domain Analysis to Evaluate AEW&C Designs

CONCLUSION

1



tory

11


• Cognitive work analysis has most commonly been used for the design and evaluation of interfaces for complex systems

• Focus in this paper is to show how work domain analysis, the first phase of cognitive analysis, can be used to evaluate design proposals for complex systems, such as military aircraft

• Traditionally, the role of human factors professionals in test and evaluation of designs has involved assessing the human performance implications of alternative design solutionsTo encourage human factors advice to be taken seriously, new steps are necessary for promoting the role of human factors practitioners in system design and development ⇒ work domain analysis offers a means for fulfill-ing this goal

• In this paper, present work domain analysis as a framework for evaluating designs in the context of military acquisition

INTRODUCTION

2



tory

11


Evaluating Designs for Complex Systems

• Complex systems have special characteristics that place unique require-ments on the evaluation of designs

① Complex systems consist of an interdependent set of human and machine com-ponents that must interact to achieve the work requirements of the system

② The work requirements of complex systems can no longer be described by a sta-ble set of task sequences or procedures

INTRODUCTION

3



tory

11


Standard Technique for Evaluating Designs

Describe standard techniques for evaluating designs and show that these techniques only partially fulfill the requirements for evaluating designs for complex systems

Technical and operational evaluation techniques: derived from systems engineering and are commonly used for military acquisition throughout the world

• The technical evaluation involves examining the physical devices of a proposed design against a set of pre-specified technical performance criteria⇒ by using this technique, evaluators develop a detailed understanding of the techni-cal solution of a proposed design

• The operational evaluation involves examining how the technical solution of a proposed design will perform in specific mission scenarios⇒ has little to offer in determining how designs will perform in a broad range of situa-tions, including changing or unanticipated conditions

INTRODUCTION

4



tory

11


A Work Domain Analysis-Based Approach for Evaluating Designs

• Work domain analysis explicitly recognizes that complex systems are sub-ject to a great many events that cannot be specified or enumerated in de-tail ⇒ WDA focuses on the fundamental functional boundaries on system per-formance and safety

• An abstraction hierarchy typically describeda. The functional purposes or high level objectives of a work domainb. The priorities and valuesc. The purpose-related functions or general functionsd. The physical functionse. The physical form

INTRODUCTION

5



tory

11


A Work Domain Analysis-Based Approach for Evaluating Designs

• The links between the layers of an abstraction hierarchy express means-ends or how-why relations

• By using an abstraction hierarchy for evaluation, the physical-device solu-tions of a proposed design can be evaluated in terms of how well they fulfill the higher-level functions and objectives of a work domain

• The abstraction hierarchy can also be used to evaluate interactions among physical-device solutions

• In summary, WDA focuses evaluation on whether an interdependent set of physical-device solutions will interact effectively to fulfill the work require-ment of a proposed system

• Thus, WDA promotes an understanding of how designs will perform in a wide variety of situations, including changing or unpredictable contingen-cies

INTRODUCTION

6



tory

11


Having outlined the theoretical motivations for a work domain analysis-based approach to evaluation

AEW&C (Airborne Early Warning and Control)– Highlights the benefits and challenges of using WDA to evaluate designs, and com-

pares it with standard evaluation techniques that were also used on this project– Complex airborne system that is currently being manufactured by Boeing for the Aus-

tralian Defense Force– The AEW&C Project Office was going to use only standard techniques to evaluate

AEW&C designs– Realized that the technical evaluation would result in a series of disparate reports

about each if the many physical devices of AEW&C– All of the physical devices of AEW&C were being designed into a single system to

support a common set of functions, priorities and values, and purposes.

⇒WDA could be used to carry out an integrated evaluation of all of the physical de-vices of AEW&C

EVALUATING DESIGNS FOR AIRBORNE EARLY WARNING AND CONTROL

7



tory

11


AEW&C Work Domain Analysis

• First step on the AEW&C project was to develop an abstraction hierarchy for AEW&C

• In reviewing the documents and interviewing subject-matter experts, used labels and descriptions of each layer of the abstraction hierarchy

• The means-ends links illustrates that evaluators can use the AEW&C work domain analysis to judge the impact of physical device solutions on the higher-level functions of the work domain


8



tory

11


• The process for using the AEW&C work domain analysis to evaluate designs took advantage of the structure of the evalua-tion team that had been set up for the technical evaluation

• The team was divided into subgroups that were responsible for evaluating the designs of particular physical devices of the AEW&C aircraft (1)

• After the subgroups had completed their reports, the head of the evaluation team, assisted by the subgroups, summarized the overall impact on the purpose-related func-tions of AEW&C (2)

• The next step was to compare the three AEW&C designs in terms of how well they supported the purpose-related functions, priorities and values, and functional pur-poses of AEW&C (3)


Using Work Domain Analysis to Evaluate AEW&C Designs

9



tory

11


Benefits and Challenges of Using Work Domain Analysis to Evaluate AEW&C Designs

Several reasons WDA was considered to be a useful approach1. The AEW&C WDA provided a set of functional criteria for integrating the results of the

technical evaluation across all of the physical devices of AEW&C2. The AEW&C WDA shifted the focus of evaluation from technical properties to the pur-

pose-related functions, priorities and values, and functional purposes of AEW&C3. Whereas the operational evaluation was restricted to six mission scenarios, the

AEW&C WDA focused evaluation on a set of functional properties that were indepen-dent of particular events

Some challenges in using WDA 4. Conveying to the evaluation team that the AEW&C abstraction hierarchy represented

the functional properties of the AEW&C work domain rather that the activity that oc-curs in the work domain

5. Standard framework for evaluation involves educating the defense acquisition com-munity in the approach


10



tory

11


• Work domain analysis provides a useful and feasible approach for evaluating designs for complex systems.

• WDA therefore complements standard systems engineering ap-proaches to evaluation in accommodating the special characteristics of complex systems

• Using WDA for evaluating designs is useful much earlier in the system life cycle for defining functional requirements and specifications

• Hence, although WDA is initially resource intensive, it can be used re-peatedly, in many powerful ways, throughout the life of a system

CONCLUSION

11

Documents

Division of Information Management Engineering User Interface Laboratory 11 Fall 09 Human Interface UI Evaluating Design Proposals for Complex Systems