AJUDY ERAU Capstone Project

AN ANALYSIS OF UNMANNED SYSTEMS CONTROL STATIONS AND AIRCREW TRAINING AS IT PERTAINS TO SAFETY OF FLIGHT

by

Aaron David Judy

A Graduate Capstone Project Submitted to the Extended Campus

in partial fulfillment of the Requirements of Degree of Master of Science in Aeronautics

Embry-Riddle Aeronautical University Extended Campus

Patuxent River Center March 2008

AN ANALYSIS OF UNMANNED SYSTEMS CONTROL STATIONS AND AIRCREW TRAINING AS IT PERTAINS TO SAFETY OF FLIGHT

by

Aaron David Judy

This Graduate Capstone Project was prepared under the direction of the candidate’s Project Review Committee Member,

Mr. Duane Mallicoat, Adjunct Associate Professor, Extended Campus, and the candidates Project Review Committee Chair,

Dr. C. J. Schumaker, Associate Professor, Extended Campus, and has been approved by the Project Review Committee. It was submitted

to the Extended Campus in partial fulfillment of the requirements for the degree of

Master of Science

PROJECT REVIEW COMMITTEE:

______________________________ Mr. Duane Maillicoat, Ed.D

Committee Member

______________________________ Dr. C. J. Schumaker, Ph.D

Committee Chair

ii

ABSTRACT

Researcher: Aaron David Judy

Title: An Analysis of Unmanned Systems Control Stations and Aircrew Training as it Pertains to Safety of Flight Institution: Embry-Riddle Aeronautical University Degree: Master of Science Aeronautics Year: 2008

While Unmanned Aerial Vehicles (UAVs) have been known around the world since the early

twentieth century, technology hindered application to present day situations. Over the past

ten years with technology advances in materials and microprocessors these UAVs have

exponential applications important to agriculture, law enforcement, air carriers, military,

homeland security, and private use. However, issues associated with Unmanned Control

Station (UCS) design and aircrew training is impacting UAV safety of flight. UAVs truly

became a part of our aviation sector during the Afghanistan war shortly after 9/11. They were

not only providing sensor capability but also began to release weapons. This has created a

paradigm shift in aviation and is reason to believe UAVs are here to provide a valuable asset

to the military and civilian sectors alike. The purpose of this study was to analyze the effects

control station design and aircrew training has upon UAV safety of flight. For the purpose of

this research, a survey was conducted on engineers, scientists, or technicians, unmanned

pilots, and manned pilots to address this problem. A literature review of current research and

the respondent’s answers were carefully evaluated and statistically analyzed to determine the

effect upon UAV safety of flight and the need for commonality.

iii

TABLE OF CONTENTS

Page

PROJECT REVIEW COMMITTEE ii

ABSTRACT iii

LIST OF TABLES vi

LIST OF FIGURES vii

Chapter

I INTRODUCTION 1

Background of the Problem 1

The Statement of the Problem 2

The Statement of the Sub-problems 2

The Study Questions 2

Limitations 2

Assumptions 2

Importance of the Study 3

Researcher’s Work Setting and Role 3

Definition of Terms 4

Summary 5

II REVIEW OF LITERATURE AND RESEARCH 7

SEARCH STRATEGY 7

UNMANNED SYSTEMS 7

Unmanned Control Stations 10

iv

Unmanned Aircrew Training 14

Summary 15

III RESEARCH METHOD 17

The Statement of the Problem 17

The Statement of the Sub-problems 17

The Study Questions 17

Research Model 17

Survey Population and Sample Size 18

Specific Treatment of the Data 18

Data Needed 18

Data Location 18

Securing the Data 18

Treatment of the Data 19

Data Analysis 19

IV RESULTS AND DISCUSSION 20

VI CONCLUSIONS AND RECOMMENDATIONS 30

REFERENCES 31

APPENDIXES 33

A DATA COLLECTION DEVICE 33

v

LIST OF TABLES

Table Page

1 UAV Levels of Control 5

2 Human Factor Issues for Hunter Accidents 8

3 Question 3 Chi Square Test 21






vi

LIST OF FIGURES

Figure Page

1 U.S. Army Hunter UAV 6 2 U.S. Air Force Predator UAV 7 3 Air Force Predator Accident Factors 8 4 Predator UCS 10 5 Survey Question 3 20 6 Survey Question 4 22 7 Survey Question 9 24 8 Survey Question 5 25 9 Survey Question 6 26 10 Survey Question 7 28 11 Survey Question 8 28

vii

viii

1

CHAPTER I

INTRODUCTION

Background of the Problem

Aircrew training and safety of flight have been a concern for aircraft since Orville and

Wilbur Wright first flew the Wright Flyer in 1903 (Sossong, 2006). Many Class A mishaps

have occurred due to inadequate aircrew training and safety of flight issues (Sossong, 2006).

This has been a known issue with manned aircraft and is currently a serious concern with

unmanned aircraft. Unmanned aircraft have been quickly fielded without addressing the need

for commonality and is beginning to hinder advanced development.

Unmanned Aerial Vehicles (UAVs) consist of a variety of vehicles both fixed wing

and rotary wing ranging from micro-size to as big as a Boeing 737 civilian transport

(Sossong, 2006). UAV operators use a variety of different Unmanned Control Station(s)

(UCS) to operate UAVs. Sometimes more than one UCS is used and the layout can differ

depending on what particular type of UAV is being operated and the flight environment. This

is a solid reason why UAV uncommon systems can limit common training and impact safety

of flight (Sossong, 2006).

Poor aircrew training and a non-standard Human Machine Interface (HMI) have a

direct impact on UAV safety of flight. Williams stated, “Estimates of the percentage of

accidents that implicate human error range from 70% to 80%” (Williams, 2004, p. 1). “In

addition, over the past 40 years, the percentage of accidents attributable to human error has

increased relative to those attributable to equipment failures” (Williams, 2004, p. 1).

Therefore, there is a strong need to develop a common UCS and develop a UAV aircrew

training method to solve poor aircrew training and the safety of flight implication.

2

The Statement of the Problem

UAVs consist of a variety of uncommon control station functions and inadequate

aircrew training which can impact UAV safety of flight.

The Statement of the Sub-problems

1. The first sub-problem. The first sub-problem is to determine the effect on how

uncommon control stations impacts UAV safety of flight.

2. The second sub-problem. The second sub-problem is to determine how inadequate

aircrew training impacts UAV safety of flight.

Study Questions

1. The first study question. The first study question is to determine how and if

uncommon control stations impact UAV operations.

2. The second study question. The second study question is to determine if aircrew

training impacts UAV safety of flight.

Limitations

Since many civil and government organizations have yet to determine the

qualifications of UAV operators or pilots, there will be a limited amount of qualified

candidates to research. However, government and civil organizations have used manned

pilots to fill this gap. This will allow the study to investigate how pilots are experiencing

uncommon control stations and inadequate aircrew training. This study should be completed

again once civil and government organizations establish solid guidelines for UAV pilots.

Assumptions

UAVs are continually increasing in size, number, and complexity. Therefore, this

study will assume the problem and results are applicable to all types of UAV systems. There

3

will also continue to be an interest in UAVs as alternatives to manned aircraft. These two

assumptions will allow the researcher to come to conclusions and recommendations worth

carrying over into future research or studies.

The Importance of the Study

While Unmanned Aerial Vehicles (UAVs) have been known around the world since

the early twentieth century, technology hindered application to present day situations. Over

the past ten years with technology advances in materials and microprocessors these UAVs

have exponential applications important to agriculture, law enforcement, air carriers, military,

homeland security, and private use. However, many UAV systems have not been thoroughly

designed neither analyzed before being presented in a real world situation with real world

operators (humans). Many UAV operators will easily say, “There is nothing Unmanned about

an Unmanned Aerial Vehicle” (Sossong, 2006). Human operators have experienced

catastrophic failures due to poor aircrew training and uncommon systems. There are many

important crew resources to consider when operating a UAV. Some of these factors include

following checklists, workload, situational awareness, fatigue, training and autonomy

(Sossong, 2006). UAVs are the next frontier in aviation similar to jet and space flight were in

the late 1950’s and early 60’s. However, if UAVs are going to provide the next leap in

aviation, the concerns with inadequate aircrew training and uncommon control station design

need to be researched and analyzed.

Researcher’s Work Setting and Role

The researcher currently has numerous experiences in UAV engineering design,

testing, and flight operations. The researcher holds a B.S. in Aerospace Engineering and a

B.S. in Mechanical Engineering. Finally, the researcher is considered a subject matter expert

4

in UAV flight test engineering at the Naval Air Warfare Center Aircraft Division, Patuxent

River, Maryland.

Definition of Terms

Autonomous. Describes a self-contained system that carries out programs or performs

tasks without outside control by acquiring, processing and acting on information (Adams,

2006).

Class A Mishap. A mishap (accident or incident) that results in loss of aircraft or

damage of one million dollars or greater, or loss of life.

External Pilot (EP). A UAV pilot external to the UCS operation, normally a type of

Radio Control pilot who manipulates the UAVs control surfaces.

Federal Aviation Administration (FAA). The federal agency responsible for the safety

of civilian aviation and enforcement of Federal Aviation Regulations.

Human Machine Interface (HMI). Where the human and the machine meet as well as

interact for a given task. Interaction can include touch, sight, sound, heat transference or any

other physical or cognitive function (Adams, 2006).

Internal Pilot (IP). A UAV pilot who controls the UAV with the UCS functions.

Mission Commander (MC). The UAV operator responsible for ensuring critical

mission functions is being prioritized within the aircraft’s flight limits and environment.

National Transportation Safety Board (NTSB). An independent federal agency

responsible for investigating civil aviation accidents (NTSB, 2007).

North Atlantic Treaty Organization (NATO). An alliance of 26 countries from North

America and Europe committed to fulfilling the goals of the North Atlantic Treaty signed on 4

April 1949 (NATO, 2007).

5

Naval Air Warfare Center Aircraft Division (NAWCAD). A Department of Navy

branch responsible for aircraft and aircraft system test and engineering.

Remotely Piloted. A pilot who controls an aircraft outside the physical domain of the

aircraft.

Standardization Agreement 4586 (STANAG). A NATO standardization agreement

that addresses the interoperability of UAVs through the UCS (Kirschbaum, 2002).

Unmanned Control Station (UCS). Provides the UAV operator with the functionality

to conduct all phases of a UAV mission. Also provides a high resolution, computer

generated, graphical user capability that enables a qualified UAV operator the ability to

control different types of UAVs and payloads (Kirschbaum, 2002).

Unmanned Aerial Vehicle (UAV). A powered, aerial vehicle that does not carry a

human operator uses aerodynamic forces to provide vehicle lift, can fly autonomously or be

piloted remotely, can be expendable or recoverable, and can carry a lethal or nonlethal

payload (Kirschbaum, 2002).

Summary

The sections above provide a background on why this research is an important one to

study. UAVs are not much different than manned aircraft when it comes to operations and

aviation rigor. The aviation community is becoming more familiar with these types of

systems and how they are impacting our aviation sector. Many organizations realize the

importance of aircrew training, common control, and the impact of safety of flight.

Unmanned aircraft obviously do not carry humans, but this does not mean dollars, resources,

and assets are worth loosing over poor implementation of aviation standards.

6

There have been many research topics written about the safety of UAVs. The

Department of Defense has pioneered the way for unmanned systems, but many of these

systems are still very immature. The following section will address how the problem

statement above ties into recently conducted research in the area of unmanned systems. The

review of literature and research will also provide more evidence towards the importance of

this study.

7

CHAPTER II

REVIEW OF LITERATURE AND RESEARCH

SEARCH STRATEGY

When researching this topic, many libraries, books, periodicals, magazines,

newspapers, and the World Wide Web were used. Key words consisted of UAV aircrew

training, human factors issues with unmanned control stations, and looking up mishap or

accidents that had occurred because of poor control station design and inadequate aircrew

training. The Embry-Riddle Aeronautical University Hunt Library Voyager and periodical

database was used to pull recent studies and results of these types of issues. Also, course

materials from the Embry-Riddle Aeronautical University Master of Aeronautical Science

program were used to provide an academic basis for conducting and analyzing the research.

The following pages and headings provide the results of this search strategy.

UNMANNED SYSTEMS

There are 5 levels of control associated with operating UAVs. Each UAV is different

in size and mission and therefore identified via a level of control. The levels of control shown

on the following page in Table 1 are based on the North Atlantic Treaty Organization

(NATO) Standardization Agreement (STANAG) 4586.

8

Table 1.

UAV Levels of Control

Level Explanation

I Indirect receipt/transmission

of UAV related payload data.

II Direct receipt of data where “direct”

covers reception of the UAV payload

communication with the UAV.

III Control and monitoring of the

UAV payload in addition to

direct receipt of data.

IV Control and monitoring of the

UAV, less launch and

recovery.

V Control and monitoring of the

UAV (Level IV) plus launch and

recovery functions.

Note. From Kirshbaum, A. (2002). NATO UAV Systems Interoperability. Unmanned Systems, 20-22. Reprinted with permission.

The most complex level of control is level V. This level not only includes controlling and

monitoring the UAV, but also launching and recovering the UAV. UAVs can be categorized

into two types: autonomous or remotely piloted. Autonomous UAVs perform missions under

control of software and a human operator controls remotely piloted UAVs

9

(Sossong, 2006). Present day UAVs are mostly a combination of both autonomous and

remotely piloted. Two specific examples are the Army’s Hunter UAV (Figure 1) and the Air

Force’s Predator UAV (Figure 2).

Figure 1. U.S. Army Hunter UAV. Note. Retrieved May 29, 2007, from http://media.primezone.com/noc/gallery/display?o=189&pkgid=2361&max=9&start=0.

Figure 2. U.S. Air Force Predator UAV. Note. Retrieved May 29, 2007, from http://www.globalsecurity.org/intell/systems/images/rq-1_011130-f-3869j-006.jpg.

10

Unmanned Control Stations

Control station design is highly dependent upon implementing the appropriate

fundamentals of human factors. Most human factor subject matter experts will agree that the

more human interaction with the machine the more likely the cause of error. The U.S.

Army’s Hunter UAV has many examples of these types of human implications. Almost 47%

of human factors related Hunter accidents occurred during External Pilot (EP) landings

(Williams, 2004). An additional 20% of the accidents occurred during takeoff (Williams,

2004). Most of the problem lies with the pilots’ perception when landing the UAV. When

the aircraft is approaching, the control inputs to maneuver the aircraft are opposite what they

would be if the aircraft were moving away from the pilot.

Display design is an issue if the right information is not being displayed at the appropriate

time or refresh rate. UAVs normally have an installed payload such as a day or night looking

camera to identify targets of interest. “Overall mission success is normally determined by the

crewmembers’ efficiency in communicating target location” (Draper, Geiselman, Lu, Roe, &

Haas, 2000, ¶ 3). Therefore, the display information is an integral part of the UAV mission.

The Internal Pilot (IP) flying the UAV from a display, keyboard, and mouse, or in some cases

a joystick, rudder pedals, and throttle controls normally gets situational awareness from the

display and voice communications with the EP and Mission Commander (MC). For example,

the Hunter UAV had an incident where a crewmember toggled off the autopilot feature, and

the information was not properly displayed. The autopilot is normally engaged when the IP is

flying the UAV, but in this case it was not and thus contributed to an accident (Williams,

2004). There are many crew errors that have contributed to this type of accident, such as

11

following the wrong procedures or in some cases the lack of procedures. Table 2 shows the

breakdown of specific Hunter human factor errors.

Table 2

Human Factor Issues for Hunter Accidents.

Issue Number Percent

Pilot-In-Command 1 7%

Alerts and Alarms 2 13%

Display Design 1 7%

External Pilot Landing Error 7 47%

External Pilot Takeoff Error 3 20%

Procedural Error 3 20%

Note. From “A Summary of Unmanned Aircraft Accident/Incident Data” by Williams, 2004, p. 5.

The U.S. Air Force Predator is also a commonly flown UAV; however the Predator is

flown all within the Unmanned Control Station (UCS) with a joystick, keyboard, rudder

pedals, and throttle controls. Also, a forward looking camera attached to the UAV provides a

3-degree field of view. Predator human factor incidents were accounted at 67% as seen in

Figure 3, much higher than Hunter.

12

Figure 3. Air Force Predator Accident Factors. Note. From “A Summary of Unmanned Aircraft Accident/Incident Data” by Williams, 2004, p. 10. The Predator has experienced many aircrew error related accidents. One error occurred

when the pilot activated a program that erased the aircraft memory. One extreme example

was the way the function keys were assigned on the keyboard. A particular order of pressing

function keys that controlled the lights on Predator was almost the same order as cutting off

the engine.

A recent example with the U.S. Customs Border Patrol Predator on April 26, 2006 shows

the concern and implications these systems can have on safety of flight. The flight was a

routine flight conducting border surveillance around the Arizona-Mexico border near

Nogales, Arizona. The flight was operating in visual meteorological conditions and an

instrument flight rules plan had been filed. The flight originated from Libby Army Airfield

(HFU), Sierra Vista, Arizona. The flight was flown from HFU by the UCS. The UCS is

configured for two operators, a pilot who controls the vehicle, pilot payload operator (PPO-1)

and a pilot payload operator (PPO-2). The controls are identical for both PPO-1 and PPO-2

(National Transportation Safety Board (NTSB), 2006). However, when control is being

13

performed from PPO-1, the controls at PPO-2 are used to control the payload or camera.

Figure 4 below shows the Predator UCS layout.

Figure 4. Predator UCS. Note. From “Unmanned Aerial Vehicles for Maritime Patrol” by Hopcroft, Burchat, & Vince, 2006, p. 6. The pilot (PPO-1) during flight reported a console "lock-up", prompting him to switch

control of the UAV to PPO-2. Checklist procedures state that prior to switching control

between the two consoles, the pilot must match the control positions at the new console as

was configured at the previous console. The PPO-2 operator stated that he did not perform

this task. Therefore, the stop/feather control in PPO-2 was in the fuel cutoff position when the

switch from PPO-1 to PPO-2 occurred (NTSB, 2006).

PPO-1 stated that after he switched control to PPO-2 the UAV was not maintaining

altitude but did not know why. Therefore he decided to shut down the UCS so the UAV

would enter its lost link procedure. The lost link procedure tells the UAV to climb to 15,000

feet mean sea level and fly a loiter pattern until link can be established (NTSB, 2006).

However, with no engine power the UAV continued to descend and ultimately crashed about

1,000 feet from nearby homes. A USA Today writer Alan Levin recently published an article

14

about the Predator crash and interviewed nearby witnesses stating, "It glided as close as 100

feet above two homes before striking the ground" (Levin, 2006, ¶5). The NTSB has only

issued a preliminary report (NTSB ID: CHI06MA121) and has yet to issue a probable cause

or any recommendations. This incident is an excellent example of how standards, human

factors, aircrew training all can affect UAV safety of flight.

Unmanned Aircrew Training

There have been two issues effecting safety of flight as seen in the examples above,

control station design and aircrew training. The human factors and Ergonomics Society

conducted an experiment on crew simulations for UAV applications and found training was a

consistent issue. Sixteen operators were studied and during the simulation indicated flight

control problems were related to poor training (Barnes and Matz, 1998). Therefore, if

unmanned pilots are not properly trained, the implication can be detrimental to flight

operations and mission success.

Currently both IP and EP UAV pilots have been mostly taken from the manned pilot

pool. There are benefits to this approach, but there are also consequences. Conventional

manned pilots are use to stick, rudder, and throttle controls. Manned pilots also do not

experience the out of the cockpit feeling resulting in a different human stimulation. Also, as

mentioned previously, manned pilots become very confused when the UAV is approaching

them in a landing pattern since the controls then become reversed. The Air Force is currently

looking at ways to better train UAV pilots, which is probably a combination of some stick

time, but more UAV flight time, to keep the operator from having to re-learn a new way of

flying.

15

The solid goal of training should be to improve mission effectiveness by minimizing

crew preventable errors, maximizing crew coordination, and optimizing risk management.

(Department of Navy, 2006). This goal should not only be applied to manned operations, but

also applied and enforced with unmanned operations. Normally, the outcome of emergencies

is dependent on the quality of teamwork the crew tries and utilizes to avoid conflicts.

Therefore, the success or failure of a team is dependent on well-laid down rules for the

activity (Sharma & Chakravarti, 2005). All of the principles of communication, situational

awareness, problem solving, decision making, judgment, leadership-followership, stress, and

interpersonal skills are also vital to effective UAV Crew Resource Management (CRM).

Formal aircrew training has helped integrate teams and more importantly help limit

and avoid fatal errors. The goal of a UAV training program should be similar to the goals of

manned operations. These goals should include, understanding the UAV operator

environment, identifying important teamwork roles to safely execute missions, and enforcing

standard operating procedures. Therefore, as explained above, there is a definitive need for a

dedicated UAV training program.

Summary

The U.S. Army Hunter and the U.S. Air Force Predator were only two examples given

on how control station design and aircrew training can impact safety of flight. There are

many different categories of UAVs; some much smaller than the Predator or Hunter, but the

same principle applies. Government or civilian organizations have yet to determine the

qualifications for a UAV pilot, aircrew training, or commonality between systems. It is

important to note, that these issues are trying to be addressed with the recently instated U.S.

Air Force UAV training school at Creech Air Force Base in Nevada. The U.S. Army has also

16

begun to setup training programs for the Hunter at Fort Huachuca, Arizona. However, other

communities have yet to determine if the U.S. Air Force or U.S. Army will establish these

guidelines or other military organizations will have an impact. Whoever establishes these

guidelines rather military or maybe the Federal Aviation Administration, the need for them is

now. Therefore, the results of this study will provide quantitative and qualitative data to help

determine an appropriate path. The study results will be extremely helpful to both policy

makers, maintainers, program offices, and more importantly the warfighter.

There are many study questions or hypotheses that could be derived from the research

shown above. However, for this study the area of importance is uncommon control station

design and inadequate aircrew training. When conducting the review of relevant literature,

these two topics continuously were brought up in many different areas of UAV research.

Therefore, the following section will address how these two issues will be researched and

analyzed.

17

CHAPTER III

RESEARCH METHOD

The Statement of the Problem

UAVs consist of a variety of uncommon control station functions and inadequate

aircrew training which can impact UAV safety of flight.

The Statement of the Sub-problems

1. The first sub-problem. The first sub-problem is to determine the effect on how

uncommon control stations impacts UAV safety of flight.

2. The second sub-problem. The second sub-problem is to determine how inadequate


Study Questions

1. The first study question. The first study question is to determine how and if

uncommon control stations impact UAV operations.

2. The second study question. The second study question is to ask and determine if


Research Model

The research model consisted of a straight forward quantitative descriptive method of

analyzing the first sub-problem and the second sub-problem. Both sub-problems data was

generated by a web survey found in Appendix A. The objective was to answer the study

questions by surveying UAV systems subject matter experts. A quantitative statistical

analysis of the results was then compounded to support the researchers’ conclusions and

recommendations.

18

Survey Population and Sample Size

The survey population consisted of three categories: Engineers, scientists, or

Technicians; unmanned pilots/operators, and manned pilots. Since there are very few

qualified unmanned pilots it is difficult to determine the total population size of this category.

Engineers, scientists, or technicians and manned pilots exceed 5,000 in population. However,

since fewer unmanned pilots are part of the population, this decreased the total population

size. Therefore to get an accurate estimate of the total population, 25 engineers, scientists, or

technicians, 25 unmanned pilots/operators, and 25 manned pilots made a total population size

of 75. Since the survey population is less than 100, the entire 75 was surveyed.

Specific Treatment of the Data

Data Needed.

The data needed to resolve the problem were the perceptions and opinions from the

three categories on how control station design and inadequate aircrew training can affect

UAV safety of flight. All of the data was carefully collected and analyzed as detailed below.

Once all data was collected it was presented in a sequential order and can be found in Chapter

IV.

Data Location.

The location of the data was contained in the responses of the web survey and the

literature review as a guide to collect the data.

Securing the Data.

Securing the data was accomplished by collecting responses from a web survey that

was given to technical individuals, unmanned pilots, and manned pilots. The web survey was

placed on the Internet and an email was sent out to the sampled individuals. The email

19

contained a web link that would direct their web browser to the survey. The questions

consisted of rating scales, agreement scales, multiple choice, and text open ended.

Treatment of the Data.

A total of 75 surveys were distributed within the community to gather a representative

pool of individuals. Each question was be identified by a number and then follow by a series

of selectable responses. To ensure confidence in the validity of the instrument tool, a pilot

survey was used. The pilot survey was administered to Senior Engineers and Chief Pilots.

The responses of the pilot survey was also tested through debriefings to ensure the survey was

producing the results needed by the researcher to conduct the study and answered both sub-

problem one and sub-problem two.

Data Analysis.

The results of the web survey were analyzed using quantitative statistical techniques.

The statistical technique used was the Pearson’s chi square test. By using this statistical

method, the study was then able to either accept or reject the hypothesis. This method also

determined in what degree the study questions answered the problem as well as determined

the distribution of the responses.

20

CHAPTER IV

RESULTS AND DISCUSSION

Sub-problem one was to determine the effect on how uncommon control stations

impacts UAV safety of flight. To solve this problem 75 surveys, included in Appendix A,

were distributed as described in Chapter III. A total of 34 surveys were completed through

Survey Monkey’s online program and analyzed. Questions 3, 4, and 9 were designed to

answer this sub-problem.

Question 3 asked the respondents to choose which UAV control functions you believe

should be associated with appropriating a UAV. Figure 5 shows the percentage of the

responses.

Figure 5. Survey Question 3. Of those that responded, 79.4% stated display, 76.5% mouse 64.7% keyboard and 52.9%

joystick. Question 3 was subjected to the chi square test, however the expected frequency

was less than 5 due to the small number of responses for throttle control and rudder pedals.

Therefore, these two categories were collapsed into joystick as well as combining the manned

pilots and unmanned pilots. Table 3 shows the calculated results. The results indicated the

21

calculated value is 5.887 and the critical value is 7.815. Since 5.887 is less than 7.815 the null

hypothesis could not be rejected.

Table 3

Question 3 Chi Square Test

Function Engineer Unmanned/Manned Pilot Total

Display 19 (27.5%) 8 (19.0%) 27

Keyboard 16 (23.2%) 7 (16.7%) 23

Mouse 18 (26.0%) 8 (19.0%) 26

Joystick 16 (23.2%) 19 (45.2%) 35

Total 69 (62.2%) 42 (37.8%) 111

chi square = 5.887; df = 3; p > .05

Question 4 asked the respondents to choose what flight critical information should be

constantly viewed by the UAV pilot or operator. Figure 6 shows the percentage of those that

responded.

22

Figure 6. Question 4.

Of those that responded, 90.9% stated location, 81.8% altitude, 81.8% airspeed, 75.8%

heading, 75.8% command and control strength, 48.5% fuel quantity, 39.4% engine

temperature, and 21.2% outside air temperature. Question 4 was also subjected to the chi

square test, but to get frequencies of less than 5, the manned and unmanned pilots were

combined. Table 4 shows the calculated results. The calculated value was found to be 1.492

and the determined critical value 14.067. The response to this question shows that detailed

performance parameters such as engine temperature and outside air temperature are not as

important as say location. Therefore, one can assume UAV dynamic parameters may not

need to be as detailed as manned dynamic parameters. This would therefore allow for a more

software driven architecture and thus decrease some workload.

23

Table 4



Altitude 14(15.6%) 13 (15.3%) 27

Heading 14 (15.6%) 13 (15.3%) 27

Airspeed 14 (15.6%) 13 (15.3%) 27

Location 17 (18.9%) 13 (15.3%) 30

Fuel Quantity 9 (10.0%) 9 (10.6%) 18

Outside Air Temp 3 (3.3%) 4 (4.7%) 7

Engine Temp 5 (5.6%) 8 (9.4%) 13

C2 14 (15.6%) 12 (14.1%) 26

Total 90 (51.4%) 85 (48.6%) 175

chi square = 1.492; df = 7; p > .05

Question 9 asked the respondents to choose why they believe operator errors occurred.

Of those that responded, 61.8% cited inadequate aircrew training, 58.8% poor control station

design, and 44.1% poor engineering design. Aircrew training and control station design are

tied very close together. Therefore the response to this question was not out of line with the

research conducted in Chapter III. Both aircrew training and control station design play a

major role in safety of flight. Figure 7 shows the responses to question 9.

24

Figure 7. Question 9.

The response for “other” was recorded as the following: errors occur because

operators are human; crew rest, fatigue, focus; loss of situational awareness; electromagnetic

interference; lack of standards and risk management. Question 9 was also analyzed using chi

square and the results can be found in Table 5. Due to the low number of responses for

unmanned and manned pilots both of these columns had to be combined so a frequency would

be greater than 5. The chi square calculated value was found to be 0.87 and a critical value of

5.991. Therefore, from this analysis we can conclude both control station design and aircrew

training play a role and impact UAV safety of flight.

Table 5



Poor Control Station Design 13 (40.6%) 9 (37.5%) 22

Inadequate Aircrew Training 13 (40.6%) 8 (33.3%) 21

Poor Engineering Design 6 (18.8%) 7 (29.2%) 13

Total 32 (57.1%) 24 (42.9%) 56

chi square = 0.87; df = 2; p > .05

25

Sub problem two was to determine how inadequate aircrew training impacts UAV

safety of flight. To solve this problem 75 surveys, included in Appendix A, were distributed

as described in Chapter III. A total of 34 surveys were completed through Survey Monkey’s

online program and analyzed. Questions 5, 6, 7, and 8 were designed to answer this sub-

problem. Question 5 asked the respondents how qualified do you feel you are to perform

UAV operations. Figure 8 below shows the responses to question 5.

Figure 8. Survey Question 5. Initially looking at the responses to question 5, it is evident that most of the

respondents do not feel very qualified to perform UAV operations. Statistical analysis similar

to how sub-problem one was analyzed was used to determine if and how aircrew training

plays a role in UAV safety of flight. Table 6 shows the calculations. To allow the expected

frequencies to be less than 5, manned and unmanned pilots were combined, not qualified /

somewhat qualified, and very qualified / extremely qualified. This provided a 2 x 2 chi

square analysis and then Yates’ correction was applied to the analysis since the frequency was

still less than 5.

26

Table 6



Not Qualified 18 (90.0%) 7 (50.0%) 25

Somewhat Qualified 2 (10.0%) 7 (50.0%) 9

Total 20 (58.8%) 14 (41.2%) 34

chi square = 4.871; df = 1; p < .05

The calculated chi square value was found to be 4.871 with a corresponding critical

value of 3.841. Therefore, since the calculated value is greater than the critical value the null

hypothesis can be rejected. This shows that most of UAV operators feel that they are

somewhat qualified. Question 6 takes another step and asked the respondents if they have

ever been formally trained on how to operate a UAV. Figure 9 below shows the responses.

Figure 9. Survey Question 6. As with question 5, question 6 was also subjected to the chi square test to determine if

the hypothesis could be accepted or rejected. Table 7 shows the calculated results. Manned /

27

unmanned were combined as well as extremely qualified and very qualified. This was done

to so the frequency would not be less than 5.

Table 7



Not Trained 12 (60%) 3 (21.4%) 15

Somewhat Trained 8 (40%) 11 (78.6%) 19

Total 20 (58.8%) 14 (41.2%) 34

chi square = 3.528; df = 1; p > .05

The chi square calculated value was found to be 3.528 and the corresponding critical

value was found to be 3.841. Therefore, since the calculated value is greater than the critical

value the null hypothesis cannot be rejected. When looking at the analysis of question 5 and

question 6, one can assume that the number of respondents who feel somewhat qualified have

also been somewhat qualified. To ensure a greater increase in UAV safety of flight, these two

groups need to be enhanced to very qualified and very trained.

Questions 7 and 8 are also related in trying to determine if the respondents were

involved in a UAV operating error and if the respondents feel if training was a cause of this

error. Figure 10 shows the response to question 7 and an overwhelming number of

respondents said No to this question, however some did say Yes. Therefore, question 8 shows

that the respondents do feel that insufficient training and experience were a cause of this error.

Figure 11 shows the responses to question 8.

28

Figure 10. Survey Question 7. Figure 11. Survey Question 8. To get a better understanding of question 8, the chi square test was subjected to the

response. The calculated results are found in Table 8. Not all of the respondents were

required to answer this question unless the respondents answered yes to question 7. The other

responses were the following: the person starting the software was unaware after the software

started, preset parameters were changed and needed to be updated; the operators did not have

sufficient manuals for conducting flight operations. The analysis in Table 8 shows the

calculated value to be 0.96 with a critical value of 3.841. Since 0.96 is less than 2.706 we

cannot reject the null hypothesis. This shows that insufficient training was the biggest factor

in aircrew error.

29

Table 8


Function Engineer Unmanned/Manned Total

Yes 1 (100%) 4 (80%) 5

No 0 (0%) 1 (20%) 1

Total 1 (16.7%) 5 (83.3%) 6

chi square = 0.96 ; df = 1; p > .05

30

CHAPTER VI

CONCLUSIONS AND RECOMMENDATIONS

Chapter IV detailed a straight forward approach to analyzing the collected data. The

survey questions were designed to address sub-problem one and sub-problem two. All the

data was analyzed using the chi square test and as can be seen in Chapter IV was in

accordance with the research conducted in Chapter II. It is evident that both UAV control

stations and UAV aircrew training play an important role in impacting UAV safety of flight.

It can be concluded that desire for both engineers and pilots is to have a UAV system

that is more reliable on software than on conventional manned aircraft controls. This was

proved by the responses to question 3 and 4. Questions 5 and 6 proved that aircrew training is

not currently efficient to reduce UAV operating errors. This analysis was backed up by

asking the respondents question 7 and 8.

A more complete analysis should be conducted within two or three years to gather if

this trend has changed. As discussed in Chapter II, the Department of Defense and other

federal agencies are trying to standardize control station requirements and qualifications for

unmanned aircrews. A larger sample size would also be desirable to gain a better

understanding of how unmanned pilots are handling currently fielded unmanned systems.

The need for unmanned systems is increasing both in the military and commercial

markets. It is imperative UAV safety of flight is addressed as more systems enter the aviation

industry. Unmanned systems are a great asset and lessons learned from manned aircraft

should be included within these studies, designs, standards and implementations.

31

REFERENCES

Adams, C. (2006). Human-machine interface. Retrieved June 2, 2007, from

http://ergonomics.about.com/od/glossary/g/man_machine.htm

Barnes, M., & Matz, M. (1998). Crew simulations for unmanned aerial vehicle (UAV)

applications: Sustained effects, shift factors, interface issues, and crew size. Santa

Monica,CA: Human Factors and Ergonomics Society.

Department of Navy (2006). Crew resource management program: OPNAVINST 1542.7C.

Retrieved February 2, 2007, from http://www.safetycenter.navy.mil

Draper, M., Geiselman, E., Lu, L., Roe. M., & Haas, M. (2000). Display concepts supporting

crew communication of target location in unmanned air vehicles. Santa Monica,CA:

Human Factors and Ergonomics Society.

Hopcroft, R., Burchat, E., & Vince, J. (2006). Unmanned aerial vehicles for maritime patrol.

Australia: Defence Science and Technology Organization.

National Transportation Safety Board. (2006). General atomics predator B preliminary report

NTSB ID: CHI06MA121. Retrieved September 12, 2006, from http://www.ntsb.gov/

North Atlantic Treaty Organization. (2007). About NATO. Retrieved May, 22, 2007, from

http://www.nato.int/issues/faq/index.html#A1

Kirshbaum, A. (2002, March). NATO UAV systems interoperability. Unmanned Systems, 20,

20-22.

Levin, A. (2006, August 6). Crash stirs debate on drone safety. Retrieved September 5, 2006,

From http://www.usatoday.com/tech/news/surveillance/2005-08-06-drone-

Safey_x.htm

32

Sharma, S., Chakravarti, D. (2006). UAV operations: An analysis of incidents and accidents

With human factors and crew resource management perspective. Aerospace Med

49(1) 2005.

Sossong, P. (2006). Crew resource management in unmanned aircraft. Naval Safety Center.

Retrieved May 1, 2007, from http://safetycenter.navy.mil/aviation/articles/CRM_

UAV.htm

Williams, K. (2004). A summary of unmanned aircraft accident/incident data: Human

factors implications. Oklahoma: Federal Aviation Administration.

33

APPENDIX A

DATA COLLECTION DEVICE

34

35

36

37

Documents

AJUDY ERAU Capstone Project