Critical Analysis Metshafe

CRITICAL ANALYSIS 1

FLORIDA INSTITUTE OF TECHNOLOGY

COLLEGE OF AERONAUTICS

Critical Analysis of

Anderson, Carolina L. (2015) "The Effects of Aircraft Certification Rules on General Aviation

Accidents," Journal of Aviation Technology and Engineering: Vol. 4: Iss. 2, Article 7.

Metshafe Samuel

AVS 5204

For

Dr. Scott R. Winter

August 2015

CRITICAL ANALYSIS 2

Critical Analysis of



Summary

The topic of safety has always been a prevalent issue in the aviation industry. Numerous

studies have been performed regarding this topic with the majority concerning human error and

organization safety practices. What the author presents in this study is a radically different

approach by examining the not so obvious effect of aircraft certification rules on general aviation

(GA) accidents. The author asserts that “The high cost of certification and installation keeps

these technologies out of small certified airplanes” (Anderson, 2015, p. 37). The author

concentrated on comparing the frequency and rate airplane accidents (Loss of control, Controlled

flight into terrain, Engine failure, Structural failure) based on each airworthiness certification

(Part 23, CAR 3, S-LSA, ELSA, E-AB). The study encompassed quantitative and qualitative

approach, with the quantitative portion consisting of a Chi-Square and analysis of covariance

(ANCOVA) statistical test. The qualitative method consisted of the author utilizing text mining

techniques to “identify sets of related words from the narrative portion of the [NTSB] report”

(Anderson, 2015, p. 39). This allowed the author to detect any patterns of similar circumstances

that occurred among the accidents.

Introduction

From the outset the purpose was concisely stated in the first sentence of the study,

allowing the reader to understand the author’s intention to analyze the frequency of GA airplane

CRITICAL ANALYSIS 3

accidents and accident rates on the basis of aircraft certification. The author precisely introduces

the topic at hand by summarizing the methods used to examine the accident frequency among

aircraft certification categories Federal Aviation Regulations Part 23 (Part 23), Civil Air

Regulations 3 (CAR 3), Light Sport Aircraft (LSA), and Experimental-Amateur Built (E-AB)

using a Chi Square test and the accident rate using ANCOVA. Furthermore, the author explained

the qualitative component of the study stating the use of “text mining techniques for the analysis

of the narrative cause of description contained within the accident reports” (Anderson, 2015, p.

33). Overall, the authors did an excellent job of outlining the problem statement and importance

of the issue at hand to the reader while conveniently summarizing the upcoming methods used in

the study.

Although the overall introduction contained a good problem statement and background,

the entire study lacked a definitive research question. The research question can aid the study by

helping the reader understand specific objectives that the author wants to accomplish regarding

the problem statement. The author could have added a research question along the lines of:

“why is aircraft certification an issue regarding safety” or “what can be done to mitigate safety

issues during aircraft certification” Despite no research question, the author provides a null

hypothesis that “there’s no significant frequency of accidents or accident rates among aircraft for

each certification in which Loss of Control (LOC), Controlled Flight into Terrain (CFIT), Engine

Failure, or Structural Failure was listed as a cause in the period between January 1, 2004 and

December 31, 2011 (Anderson, 2015, p. 38). For the variables, the author did not explicitly

define which is what variable, but the reader can easily identify the respective variables. The

independent variables are the type of aircraft certification and the dependent variables are the

listed cause of accidents (LOC, CFIT, Engine Failure, Structural Failure). The plethora of

CRITICAL ANALYSIS 4

variables can cause induce confusion for the average reader who might not be familiar with

aviation concepts. If the author explicitly describes which variable is what in a graphical format

to accompany the text, then it can enhance the readability of the introduction without getting lost

in the details.

The literature review in the study is plentiful but limited in scope. The majority of

references are from government agencies including FAA, NTSB, GAJSC, etc. The references

regarding FAA are appropriately used when supporting various arguments related to the

regulatory environment or for details regarding relevant data highlighting general aviation

statistics. The majority of the referenced sources are used in an annotative nature, by briefly

backing up assertions and arguments as well as many instance where the author simply states

“based on literature” when the reader has no point of reference regarding that statement. Critical

analysis of multiple sources is not evident, nor are the sources compared and contrasted.

Although there’s an apparent lack of critical review of other literature in the study, the study

itself is not short any highly detailed information that is well researched and extracted from the

various references.

Methodology

The methodology section of the article is very comprehensive and gives the reader a

glimpse into every facet of the research approach, design, and procedure. The author

highlighted the major characteristics of the target population by elaborating that the study will

focus on analyzing GA accidents involving fixed-wing, single-engine, reciprocating-piston

aircraft and gave good reasons on why that particular population was chosen with a few reasons

CRITICAL ANALYSIS 5

being, piston airplanes account for 67% of the total civil aircraft population, 64% of GA

accidents involved personal flying in fixed-wing airplanes, and 78% of personal flying was

conducted in fixed-wing, single-engine piston airplanes (Anderson, 2015, p. 38). The sample

chosen consisted of accidents of single-engine, fixed-wing airplanes with a maximum

certificated takeoff weight of less than 12,500 pounds between January 1st, 2004 and December

31st, 2011. The author’s decision of the time period was “based on the creation of the Light Sport

Aircraft (LSA) category in 2004 and the end time was based on the availability of NTSB

accident reports with a probable cause” (Anderson, 2015, p. 39) Overall the author did very well

in describing the major characteristics of the population and sample selection.

For the instrument, the author does not give the reader general information on what

“instrument” is used but rather describes the use two statistical tests as well as text mining

technique to detect patterns in the NTSB narratives. The two statistical test are comprised of the

Chi-Square and the ANCOVA test. The author fails to adequately describe the reason and basis

for the choice of these statistical tests, but rather leaves it to the reader to deduce the rationale

behind it. Aside from the description and use of these statistical tests, the partial information on

the choices of the test and “instrument” explanation makes it rather hard for the reader to

comprehend the reasons for the actions taken in the study.

For the procedure the author starts with the Chi-Square test to compare the frequencies of

accidents among airplanes certified under Part 23, CAR 3, S-LSA/E-LSA, or E-AB in which

LOC, CFIT, Engine failure, or Structural Failure was listed as a cause from 2004 to 2011

(Anderson, 2015, p. 39). If the test value had a probability of less than .05, then the null

CRITICAL ANALYSIS 6

hypothesis would be rejected. The author then moves on to the second statistical test which is

ANCOVA. The test will compare the accident rates among airplanes certified under Part 23,

CAR 3, S-LSA/E-LSA, or E-AB in which LOC, CFIT, Engine failure, or Structural Failure was

listed as a cause from 2004 to 2011 (Anderson, 2015, p. 39). If the F-statistic in the ANCOVA

test had a probability of less than .05, then the null hypothesis would be rejected. The author

calculated the accident rate in the ANCOVA test by dividing the number of accidents per year

for each of the aircraft airworthiness categories by the number of hours flown per year for each

category. For the qualitative portion of the study, the author analyzed the narrative cause section

of the NTSB reports for accidents in which LOC was listed as a cause. The author cites that LOC

accidents are of special interest because they are the main cause of GA accidents. The author

further cites that “text mining [in the qualitative portion of the study] can help identify sets of

related words from the narrative cause portion of the report; it can also identify clusters of

similar circumstances, possible patterns, and relationships among accidents” (Anderson, 2015, p.

39).

The data used in the study was obtained by the author from the publicly accessible NTSB

accident database that was available online in Microsoft Access format and the FAA type

certificate database. The sample size in the study was provided in the study as well as an

adequate rationale behind the decision as mentioned earlier.

Results

The results section of the study is structured in an informative manner with the tables

neatly organized. The results of the Chi-Square and ANCOVA statistical tests were included in

CRITICAL ANALYSIS 7

the tables. The author adequately describes the frequency and descriptive data presented in each

graph. The author went to great lengths to test and evaluate 8 hypotheses: (4 tests for Chi-Square

& ANCOVA). The 4 tests represents each of the 4 listed accident causes (LOC, CFIT, Engine

Failure, Structural Failure). Additional uses of bar graphs and other statistical data from external

sources from the FAA brings more insight into the findings and allows readers to digest the

information better with good visuals of the data. From the hypotheses testing, the Chi-Square

tests indicated that there was a significant difference in the frequency of accidents in which LOC

and Engine Failure was listed as a cause. Additionally, the ANCOVA test indicated that there

was a significant difference in the accident rate in which Engine Failure was listed as a cause.

Furthermore, the author’s text mining analysis found a group of clusters or words associated with

LOC during takeoff/landings, maneuvering flight, and degraded weather. Accidents involving

the Part 23 category had a cluster of words associated with spatial disorientation and instrument

flight. The CAR 3 category had a cluster associated with low visibility, poor weather conditions,

night, and VFR flight into IMC conditions (Anderson, 2015, p. 44).

Conclusions

The author revisits the results of hypotheses and the text mining analysis in order to

indicate that GA certification rules do not have a statistically significant effect on aircraft

accidents except on frequency of LOC accidents and on the frequency/accident rate of Engine

Failure. The author states that “with respect to LOC accidents, government oversight could have

become an obstacle in the implementation and installation of new safety enhancing equipment

into old aircraft that could possibly reduce the number of LOC accidents” (Anderson, 2015, p.

45). Due to the author’s unique approach to this issue, there is no study out there the is

CRITICAL ANALYSIS 8

comparable. The author also pointed out certain flaws in government oversight that needs to be

improved regarding the certification process of E-AB airplanes.

The author offered more recommendations such as the adoption of consensus standards

for GA aircraft certification, which would allow simplification of retrofitting legacy fleet of GA

airplanes with the installation of safety enhancements in an economically viable manner. These

suggestions underscored the importance of loosening stringent and costly certification processes

in order to allow GA aircraft (which the majority are really old) to have the latest safety

technology.

References



CRITICAL ANALYSIS 9

Documents

Critical Analysis Metshafe