Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
1
Running head: GOOD EDUCATION RESEARCH
A consideration of ethics, methods and theory:
Is there consensus on what constitutes good education research?
Kelly. D. Bradley∗, Kenneth D. Royal, Jessica D. Cunningham, Jennifer A. Weber,
Jennifer A. Eli, Tara Baas, and William E. Harris, Jr.
University of Kentucky
Special Thanks to Eleanora Byrd, University of Kentucky, for survey management
∗ Use Kelly D. Bradley, Ph.D. as contact: 131 Taylor Education Building, Lexington, KY 40506; [email protected]
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
2
Abstract
The question of what constitutes good education research has received a great deal of attention.
Even though as a whole, purpose, theme and methods vary across research agenda, it seems that
the education community could have some consensus on what constitutes good research. For the
purpose of this study, a survey investigating faculty and graduate student perceptions of good
education research was administered in the College of Education at a large Southeastern
university. The primary goal of the study was to reveal which characteristics are most frequently
endorsed by members of various disciplines within education research. Results suggest that items
connected to ethics and theory were empirically found to be relatively easy to endorse for the
majority of participants. When connected to methodology, variation in perceptions was
noticeable.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
3
A consideration of ethics, methods and theory:
Is there consensus on what constitutes good education research?
Faculty and graduate students are inundated with and trained to be well-versed in
research methods, theory, ethics and other common elements associated with the research
process. In both the classroom and the profession, numerous ways to conduct research exist.
Some education researchers prefer employment of quantitative methods, qualitative methods, or
a combination of the two approaches. Regardless of the approach and form of inquiry, it is
reasonable to assume that common ground and variation in the perceptions of what constitutes
good education research can be identified.
Connecting to the general theme of standards in conducting research in education, this
paper sets out to explore the question of what constitutes good education research. Inspired by an
article published in the Educational Researcher (Hostetler, 2005), this discussion is centered on
the results of a survey about perceptions of good research completed by College of Education
(COE) faculty and graduate students at a public southeastern university. Using the American
Educational Research Association (AERA) and the American Psychological Association (APA)
as the guiding sources, statements about good research were derived from the organizations’
guidelines and developed into survey items. The results of the survey along with the existing
literature base will serve as a foundation for operationalizing good education research.
As the survey instrument is in the early stages of construction, a Rasch measurement
model was employed to evaluate the quality of the survey instrument in terms of rating scale
appropriateness and clear divisions between established constructs. A hierarchy of survey items
is presented to illustrate endorsability of characteristics of good research. The findings are
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
4
discussed in three overarching areas: ethics, methods and theory. The paper stimulates dialogue
about these areas and how each functions as a driving force for education research.
Theoretical Framework
AERA and APA were chosen as the guiding authorities on the topic of good education
research as both govern and direct research and practice in the domains instructed throughout the
college of education. To better inform and guide the research, an overview of the literature as it
pertains to the three major domains ethics, methods and theory is presented, specifically
literature relating to perceptions of good research. Also, a brief overview of the Rasch model is
presented to inform the methods and results which follow.
Ethical Considerations
Hostetler (2005) argues that recent debates regarding good research and its relationship to
ethics can be limited to methodology. The bulk of literature regarding ethics and good research
seem to support this assertion. Social science researchers tend to agree about what is and is not
proper when conducting scientific inquiry (Babbie, 2008). Several ethical considerations have
been highlighted in the literature and are investigated by this study.
Numerous ethical considerations involve participation, consent, and dissemination of
results. Concerning participation, research volunteers must participate in studies on their own
free will (Nardi, 2003) without being forced or otherwise coerced (Babbie, 2008; Simmerling,
Schwegler, Sieber, & Lindgren, 2007). Participants should also be granted confidentiality to
prevent the linking of personal identity to various responses; and unless stated otherwise,
participants should remain anonymous as well (AERA, 2002). Further, participants should
always be granted informed consent. That is, participants should be made aware of their rights
and informed of any risks or possible repercussions that could result from their participation in
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
5
the study. Typically, Institutional Review Boards (IRB) monitor related research ethics to ensure
appropriate measures are taken to protect potential participants. Similarly, a number of issues
relating to the dissemination of research findings are discussed in the literature.
According to Babbie (2008), researchers are ethically obligated to admit unexpected
findings and the limitations of their work. The AERA encourages researchers to report their
findings without selectivity and/or secrecy. That is, researchers should objectively report results
and refrain from distributing only the results that are deemed favorable to their personal beliefs.
Likewise, researchers are obligated to report any results that may appear unfavorable. Although
the language used varies slightly, the literature regarding ethics and methodology is generally
grounded in these ethical guidelines. Beyond the topic of ethical guidelines and methodology,
the literature begins to diverge into small segments regarding the ethical context of research
concerns.
The literature addresses the changing role of researchers utilizing qualitative
methodology, including researcher obligations to examine the ethics and morality of their work
and to explicitly acknowledge the fragility of the relationship between the researcher and those
being researched (Ebbs, 1996). In a similar vein, the researcher/researched relationship has
become controversial in the context of “outsider research.” The argument centers on the
epistemological claim that researchers who are not members of disempowered groups should not
attempt to study these groups. The outside researcher cannot accurately understand or represent
such groups and faces an ethical dilemma of exploiting or being disrespectful of participants
(Bridges, 2001).
The literature also addresses issues pertaining to researchers’ expectations and ethical
standards. Professional ethics in academe can be viewed as cultural, personal, and academic, but
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
6
are often evaluated situationally, thus making it difficult to find an extensive yet accessible code
of conduct for academics (Bruhn, Zajac, Al-Kazemi, & Prescott, Jr., 2002). Another dilemma
facing researchers is the origin of ethical guidelines for researchers, The Belmont Report.
However, the report’s emphasis on medical issues does not seamlessly transfer into the social
and behavioral science environments (Simmerling, et al, 2007).
Ethical considerations in research writing are often internalized, thus the visibility of
these components of research can be masked in publications. Authors are limited by criteria
imposed by journal editors regarding structure and content. The evidentiary condition of
arguments in research literature can be viewed as impeding writers’ prerogatives to address
ethical implications or considerations of their work in manuscripts. The nature of research
articles may not allow the moral dimension of research to be explicitly noted in journals
(Kansanen, 2003). Discussions pertaining to the implicit nature of ethics in research have led
researchers to address a need for improved ethics education. Researchers have advocated that
scientific societies offer ethics education programs to members and graduate students (Iutcovich,
Kennedy, & Levine, 2003), and have argued for more comprehensive ethics training in classes
(Muskavitch, 2005).
Although the generally recognized guidelines have been presented, work connecting the
quandary of research and ethics to the need for improved ethics education is not apparent. A
body of literature related to how researchers navigate ethical dilemmas, as well as how they
conceive the role of ethics in research beyond methodology, would be beneficial in making the
argument that stronger ethics education would (or would not) lead to better research practice.
Methodological Considerations
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
7
Educational research is “the collection and analysis of information on the world of
education so as to understand and explain it better” (Opie, 2004). Thus, “high-quality” education
research would not only involve using carefully constructed designs, but also implementing the
most appropriate methods of data collection and analysis with the purpose of answering research
questions aimed at understanding the world of education. More often than not, “high-quality”
research concludes as debate on a methodological level. Methodological debates in education
research and thoughts on the nature of knowledge and learning play a vital role in how one
determines what is or is not “high-quality” education research (Hostetler, 2005).
For the purposes of this paper we make a distinction between methods and methodology.
Methods are the techniques and/or procedures for data collection and analysis. A methodology
refers to a philosophical framework, strategy, or plan of action implemented with the intention of
obtaining knowledge by considering the best methods or procedures leading to data that will
provide evidentiary basis for what is being studied (Creswell, 2003; Creswell & Plano-Clark,
2007; Sikes, 2004). “Methodology is concerned with the description and analysis of research
methods rather than the actual, practical use of those methods” (Sikes, 2004, p. 16). For example,
the methodology for this project is survey research as it is a “strategy or plan of action that links
methods to outcomes governs our choice and use of methods” (Creswell, 2003, p.5). The method
used for data collection is a Web-based survey and the method for data analysis involves
implementation of the Rasch model.
Two major factors influence a researcher’s choice of procedures and methodology when
conducting educational research: 1) his/her philosophical assumptions about reality, knowledge
and values; and 2) the research question(s) under investigation. Educational researchers
inherently bring to the table certain philosophical assumptions that guide their approach to
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
8
conducting educational research (Creswell, 2007). In particular, their views on the nature of
reality (ontology), the nature of knowledge (epistemology), and the nature of human values
(axiology) play a crucial role in how one carries out research (Sikes, 2004). Researchers might
ask themselves the following questions to ascertain underlying philosophical assumptions that
guide their approach to research: Is there a single reality or multiple constructed realities? Is the
relationship between the researcher and the subject independent or dependent? Is inquiry value
free or value bounded? Tashakkori and Teddlie (1998) outline four philosophical orientations
that represent the major stages of the paradigm wars prior to World War II and up to the 1990s:
logical positivism, post positivism, constructivism, and pragmatism. Consider these
philosophical orientations along a continuum with logical positivism on the far left and
constructivism on the far right. Logical positivists believe in a single reality; the relationship
between researcher and subject is independent; knowledge is objective; inquiry is value free,
context-free generalizations are possible; and causal links can be isolated and identified.
Quantitative research methods are most often identified with the logical positivism philosophical
orientation as these methods propose measuring and analyzing causal relationships within a
framework that is value-free. Quantitative research methods generally involve working with
numerical data collected from a representative sample and analyzed using statistical methods.
Likewise, with qualitative research, the researcher and the subject are seen as independent;
knowledge is viewed as objective; and methods use deductive reasoning.
Post positivism became popular after World War II. Unlike logical positivists, post
positivists believe that reality is constructed; knowledge is objective; inquiry is influenced by the
values of the researcher; context-free generalizations are possible; causal links can be isolated
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
9
and identified; and strive for separation between researcher and subject while acknowledging
their own biases
Moving into the 1960s, the emergence of constructivism can be seen as lying to the far
right of the continuum. Constructivists believe in multiple constructed realities; the relationship
between researcher and subject is dependent; knowledge is negotiated; inquiry is value-bounded;
and context-free generalizations are not possible. Qualitative research methods are most often
identified with the constructivist philosophical orientation as these methods involve analyzing
descriptive data using inductive reasoning to draw inferences.
The fourth major stage, pragmatism, focuses on practicality emphasizing a “what works”
best approach. Pragmatists believe in singular and multiple realities (Creswell, 2007).
Pragmatists take “multiple stances” when it comes to axiology as they would “include both
biased and unbiased perspectives” (p.24). The pragmatist paradigm allows for the use of mixed
methods in educational research, that is, researchers can “combine the qualitative and
quantitative approaches into the research methodology of a single study or multiphase study”
(Tashakkori & Teddlie, 1998, p. 18).
To summarize, quantitative and qualitative approaches to educational research have been
distinguished and thereby defined on the basis of the type of data used (numeric or descriptive),
the mode of analysis (statistical or interpretative), the logic employed (deductive or inductive),
the type of investigation conducted (explanatory or exploratory), focus of the research (quality or
quantity), as well as on the basis of underlying paradigms (positivism or constructivism)
(Tashakkori & Teddlie, 1998; Merriam, 1998; Creswell & Plano-Clark, 2007). A combination of
qualitative and quantitative methods is regarded as mixed methods. As argued in the literature:
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
10
A mixed methods study involves the collection or analysis of both quantitative and/or
qualitative data in a single study, in which the data are collected concurrently or
sequentially, are given a priority, and involve the integration of the data at one or more
stages in the research process (Creswell, Plano-Clark, Gutmann, & Hanson, 2002, p.
212).
The second major factor influencing a researcher’s procedures and methodology is the
research question(s). “The important issue is to ensure that the research procedure(s) used is
appropriate to the research question being asked and the research answers being sought” (Opie,
2004, p.9). “Research methods should follow research questions in a way that offers the best
chance to obtain useful answers” (Johnson & Onwuegbuzie, 2004, p.18). Once a researcher has
selected a methodology, one must be able to justify or provide a rationale for choosing a
particular approach or procedure that he or she feels will best address the research question. “It is
on the match between methodology and procedures and research focus/topic/questions that the
credibility of any findings, conclusions and claims depends, so the importance of getting it right
cannot be overemphasized” (Sikes, 2004, p. 17).
Theoretical Considerations
Kerse and Elley (2003) believe defining good research is as subjective as defining beauty,
as beauty lies in the eyes of the beholder. Brandon (2000) has simply defined “good” research as
any project producing useful results whether positive or negative. Interestingly, Brandon posits,
“Research is frequently frustrating and often leaves us more confused than when we started” (p.
1). In multiple texts and articles, many experts in the field of research have referred to research
as a “science”. Whitley (1996) describes science as the “systematic process for generating
knowledge about the world… consisting of three important aspects; the goals of science, key
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
11
values of science, and perspectives on the best way in which science can go about generating
knowledge”(p. 2). Making this reference presumes a great deal of meticulous efforts take place
when research is performed. Additionally, scholars could speculate about the driving force
behind quality research as a science.
One factor which many consider essential to “high quality” research is theory. Multiple
researchers and educators (Brandon, 2000; Elmes, Kantowitz, & Roediger, 1995; Hughes, 1999;
Kerse & Elley, 2003; Mikk, 2006; Nardi, 2003; Whitley, 1996) have noted the importance of
theory in quality research. Hughes (1999) attested that every piece of research should have
definite connections to some theory or existing body of literature. The most common strategy of
connecting current research to theory is through a literature review (Nardi, 2003). Nardi stated an
in depth review of literature provides an introduction of theories, which leads to a formulation of
questions and concepts used to test current theories or create new ones.
Scholars have often labeled theory as the building block of quality research (Brandon,
2000; Hughes, 1999; Mikk, 2006; Nardi, 2003). Hughes (1999) believes without links to
previous theory, researchers become suspect about the possibility of repeating earlier studies.
When faculty, students, and private investigators begin the early stages of their research, they
rely upon existing theories to help guide their studies (Brandon, 2000). According to Mikk
(2006), theory allows the researcher to establish a hypothesis in order to answer the research
question. So, how do researchers define theory?
According to Nardi (2003), theory is defined as “a set of statements logically linked to
explain some phenomena in the world around us” (p. 27). Whitley (1996) defines theory as “a
set of statements about relationships between variables” (p. 9). He continues to characterize
variables as abstract constructs (e.g., memory) or concrete objects (e.g., length). Some
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
12
researchers have alluded to judging the relevance of a theory (Elmes, Kantowitz, & Roediger,
1995) by the larger number of occurrences or observations of a theory or the fewer number of
statements of a theory implying a better theory. But, this is not always true in every situation.
Elmes, Kantowitz, and Roediger (1995) and Whitley (1996) have discussed the two
major functions of theory in research practice as organization and prediction. First, theory
provides a framework for the “systematic and orderly” (Whitley, 1996, p.7) presentation of data.
Second, theory allows researchers to introduce predictions for occurrences for which no data is
collected. Whitley also introduced a third major function of theory in research practice – as
theory organizes data, it also extends the current knowledge in the field. In other words, data
from theory-driven research can modify current theories, establish linkage to other theories,
allow the merging of theory through a convergence of concepts, and extend knowledge across
disciplines.
Many researchers have acknowledged the important relationship between theory and
research (Nardi, 2003; Whitley, 1996). This relationship has fashioned two forms of rationale -
deductive and inductive reasoning. Deductive reasoning utilizes hypotheses from a theory to test
that theory. On the other hand, inductive reasoning uses data from previous tests of theory to
verify or modify certain aspects of the theory when results fail to provide support. Nardi states:
“Most research involves both processes: a review of literature tells us what theories and
explanations we can use to deduce specific research questions that are then used to get
data that form the basis for inducing or modifying a theoretical perspective to explain
what was observed and measured” (p. 28).
In an attempt to summarize theoretical considerations, theory has been defined as the
building block to quality research. Researchers rely on theory to help organize data and to
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
13
formulate new hypotheses for their current research. Further, researchers often link previous
theories with current theories, or in some cases, create entirely new theories. Observing the
importance of theory in quality research has created theory-based practices, instrument
development, and other applications relevant to educators and researchers. One must not forget
the importance of theory in the formation of research hypotheses. With this in mind, the
expectation for high-quality research is to be theoretically grounded or linked to previous
research.
An introduction to the Rasch model
The Rasch model is a one-parameter logistic model within item response theory (IRT) in
which the amount of a given latent trait in a person and the amount of that same latent trait
reflected in various items can be estimated independently yet still compared explicitly to one
another. Given the Rasch model follows mathematically from the requirement of invariance of
comparisons among persons and items, a Rasch analysis is appropriate when the total score on a
questionnaire is used to make inferences about an individual’s level of a latent trait inherent in
that individual. Classical Test Theory (CTT) also uses the total score to characterize each person,
but it asserts the total score as the relevant statistic with little consideration of anomalies in the
items or the respondents. Applications of the Rasch model account for these anomalies and
provide a more informative score (Andrich & Luo, 2003).
The Rasch model, introduced by Georg Rasch (1960), yields a comprehensive picture of
the construct under measurement and the respondents on that measure. It allows observations of
respondents and items to be connected in a way that indicates the occurrence of a certain
response as probability rather than certainty and maintains order in that the probability of
providing a certain response defines an order of respondents and items (Wright & Masters,
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
14
1982). The model is static, meaning that for each person having a certain amount of the given
latent trait; the model specifies the probability of a response in one of the categories of an item.
The response process is a classification into ordered categories defined by thresholds, the point
where the probability of a response in either one of two adjacent categories is 50%, analogous to
the markings on a ruler (Andrich & Luo, 2003; Guttman, 1950).
The Rasch model uses the sum of the item ratings simply as a starting point for
estimating probabilities of those responding to each item. In the case of a questionnaire,
probabilities are based upon individuals’ willingness to endorse a set of items and the difficulty
to endorse those items. The difficulty to endorse is assumed to be the main characteristic
influencing responses (Linacre, 1999). Rasch analysis reports both person ability (i.e.,
willingness-to-endorse) and item difficulty (i.e., difficulty-to-endorse) estimates along one logit
(log odds unit) scale, “a unit interval scale in which the unit intervals between the locations on
the [combined person-item scale] have a consistent value or meaning” (Bond & Fox, 2001, p.
29). Bond and Fox explain that employing Rasch techniques allows for the ordering of
respondents along this continuum of ability or willingness to endorse items, and orders items
along a continuum of difficulty to endorse. Rasch measurement is relevant whenever a
questionnaire or assessment is constructed to measure the degree of some property inherent in
persons or other entities, as is the case in this study.
Purpose
The purpose of the study was both exploratory and evaluative in nature. Data collected
via the Web-based survey are intended to help investigate characteristics that faculty and
graduate students in the college of education associate with good education research as well as
evaluate the use of the online survey. The primary expectation of the study was to reveal which
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
15
characteristics are most frequently endorsed by members of various disciplines within
educational research. The secondary expectation was to find support for the quality of the Web-
based measure employed. Initial data collected in the pilot study using the Web-based
measurement tool provide guidance for revisions of the instrument, which will undergo similar
subsequent analysis to assess the validity of the revised survey. Specifically, the objectives of the
study were to: 1) illustrate an overall picture of participants’ characterization of “good” or
“quality” research; 2) examine differences in preferences based on participant demographics, and
3) validate the Web-based survey to be used for data collection.
Research Questions
The data collected in the study were used to help answer questions such as:
1) What do graduate students and faculty consider to be “good” or “quality” educational
research?
2) What differences exist in participants’ endorsement of “quality” research characteristics
based on participant demographics?
3) Is the survey a valid and quality method of data collection regarding student and faculty
impressions of educational research?
Methods
Response Frame
Two primary populations have been targeted for this study. The sampling frame includes
faculty (85) and graduate students (approximately 800) from the College of Education (COE) at
a public southeastern university. The university’s COE includes undergraduate- and graduate-
level programs in six areas of educational research and practice, including: curriculum and
instruction (i.e.: teaching and learning); educational leadership studies; educational and
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
16
counseling psychology; educational policy and evaluation; kinesiology and health promotion;
and special education & rehabilitation counseling. All respondents were expected to be at least
21 years of age. No criteria were provided for exclusion, as all COE faculty and graduate
students were surveyed. The foreseen demographic profile anticipated for the study participants
was conceptualized based on 2006 Headcount Enrollment data provided by the university’s
Office of Institutional Research, which expected the student sample should consist of
approximately 72% female and 28% male; 77% White/Caucasian, 10% African American, and
the remainder of the sample consisting mostly of Asian, Hispanic, and nonresident alien students.
Procedure
A formal request to the Registrar’s office was sent requesting all College of Education
faculty and graduate students’ email addresses. Concurrently, an email notice was sent via COE
departmental LISTSERVs to inform potential respondents that the survey should arrive in their
email inbox in the near future. The sampling frame received an email invitation to participate in
the survey, which was embedded with the link to the survey. The email message also contained
statements regarding the following: purpose of the survey; informed consent; voluntary nature of
the survey ensuring no penalties or repercussions of any kind regardless of whether or not a
person chooses to participate; confidentiality; aggregation of data; amount of time anticipated to
complete the survey; timeline of when the survey will be closed; and contact information should
potential respondents have any questions and/or concerns.
Responses were collected and stored on a secure web server provided by the Web-based
survey software program SurveyMonkey (2007), which is hosted on secure servers. Once the
survey was closed, data were downloaded onto the primary investigator’s computer. From there,
data were burned onto a CD and stored in a locked file cabinet in the principle investigator’s
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
17
office. With regard to data analysis and dissemination of results, no personally identifiable data
are presented. All data appear in aggregate form.
Instrumentation
The survey instrument consisted of 60 questions partitioned into two components, namely
statements describing potential characteristics of good education research (39) followed by a
demographics component (21). Statements of good education research were grouped into three
domains: methodological (17), ethical (10), and theoretical (12). All survey questions were
written or adapted from guidelines for “quality” or “good” research provided by various research
organizations which lead and govern research in the given educational disciplines. Namely,
questions resemble guidelines set forth by the American Educational Research Association
(AERA), American Psychological Association (APA), and National Council of Teachers of
Mathematics (NCTM). The demographic component of the instrument included items which
encompass both the individual and professional background of the participant. These data were
designed to assist in determining if any differences exist in participants’ endorsement of
“quality” research based on various demographic criteria. All questions were devised, reviewed
and revised by the research team developing the survey.
Analysis
Crucial to determining common characteristics of ‘good’ education research is the quality
of the instrument used to examine participant responses. Bond and Fox (2001) argue,
“…interpretation of analyses can only be as good as the quality of the measures” (p. 26). Prior to
the full scale administration of the survey instrument, the quality of the instrument must be
assessed. In an effort to ensure measurement stability, this study employs a one-parameter Item
Response Theory model, otherwise known as the Rasch model.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
18
An overall partial credit model was applied with persons and items to test the overall fit
of the data to the model using Winsteps version 3.57.4 software. The mathematical expression
employed for constructing measures through responses is log (P nik / P )1( −kni )/B n –D ik (Andrich,
1978) where P nik represents the probability the person n when responding to item i would be
observed in category k; similarly P )1( −kni ; B n is the attitude of person n; D ik is the difficult of
item i with the impediment to being observed in category k relative to k-1. The partial credit
model, mathematically equivalent to the rating scale model within Winsteps, was chosen due to
the possibility that respondents may interpret the scale differently depending on the item.
Missing data were treated as missing since it was reasonable to believe respondents might not
wish to answer all survey items with integrity. Further, it was decided that imputing means or
other substitutes for missing data would be inappropriate since missing data are not problematic
with the Rasch model and could result in data and information not reflective of the answers
provided by the respondents.
First, the Rasch model was employed to guide revisions to the survey instrument prior to
the full scale study. Survey items and respondents not adequately fitting the model requirements
were identified using the mean square fit statistics, with a reasonable range determined within
one standard deviation of the average mean square fit statistic (Wright & Stone, 2004).
Inspecting the outfit mean-squares provides evidence about the fit of the data to the model. The
infit mean-squares are used to determine the fit of the item within the construct. The guidelines
outlined by Linacre (2004) were used to evaluate the rating scale category effectiveness based on
the responses. Point-biserial correlations were inspected to investigate the orientation of the
latent variable to ensure that the polarity of the items were of the same sign, or direction (i.e. all
point-biserial correlations were positive). The number of observations and the distribution of
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
19
observations across categories were examined to describe the functioning of the rating scale
categories. Probability curves were used to visually inspect the rating scale category function.
Advancing average measures with each category and step calibrations ensure the rating scale
measure is stable and accurate.
Finally, Rasch results and descriptive statistics were used to present findings from the
study. Relative frequency tables illustrated a general picture of the characteristics the participants
associate with good education research. The item map produced from the Rasch analysis
provided an empirical hierarchy, which represents characteristics of good research that are most
likely to receive endorsement.
Results
Demographic information for participants
The sample for this study consisted of 126 faculty (n = 50; 39.7%) and graduate students
(n = 76; 60.3%) from UK’s College of Education. Below are personal demographic
characteristics for both faculty and graduate students. First, Table 1 examines faculty
demographic characteristics:
[Insert Table 1 Here] Graduate students also comprise a significant portion of this sample. Of the 76
respondents, 50 (69.4%) reported full-time status, 14 (19.4%) part-time status, and 8 (11.1%)
inactive. Table 2 examines graduate student demographic characteristics:
[Insert Table 2 Here] With regard to the collective sample, 89% (n = 105) of respondents reported being of
White/Caucasian ethnicity. Asian and Pacific Islanders comprised 5.9% and African American
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
20
or Black respondents accounted for 5.1%. A plethora of areas of academic interest were reported
from the sample. The results are presented in Table 3 below.
[Insert Table 3 Here]
Respondents were asked several questions to further detail their preferences regarding
research (see Table 4). When given the forced-choice question to indicate a primary interest in
either teaching or research, 43.4% of faculty and students reported “very heavily” or “leaning
toward” research. Respondents were also asked to estimate the number of hours spent on
research in a given week. The majority of respondents (46.0%) reported spending one to nine
hours on research each week, followed by approximately one fourth (24.2%) spending 10-19
hours each week on research. Faculty and students were also asked to report their primary
research methodology, where 40.7% reported mostly or entirely quantitative methods. When
asked about their perceptions of educational research quality, the majority (57.5%) reported
average quality. For complete results, see Table 4 below.
[Insert Table 4 Here]
Fit of data to the model
First, person and item reliability estimates are important in determining fit of the data to
the model. With a total of 137 participants measured, the person reliability and separation are
0.86 and 2.48, respectively. For 39 survey items, the item reliability and separation estimates
were 0.98 and 7.64, respectively. Point-biserial correlations were inspected to investigate the
orientation of the latent variable to ensure that the polarity of the items were of the direction (i.e.
all point-biserial correlations were positive), which was the case.
Survey items and respondents not adequately fitting the model requirements were
identified using the mean square fit statistics, with a reasonable range determined within one
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
21
standard deviation of the average mean square fit statistic (Wright & Stone, 2004). Inspecting the
outfit mean-squares provides evidence about the fit of the data to the model. The infit mean-
squares were used to determine the fit of the item within the construct.
Misfit is indicated by an item’s infit mean square fit statistic above 1.23, while overfit is
indicated by an infit mean square fit statistic less than 0.79. An infit mean square above 1.23
illustrates that the item does not fit according to the response patterns of other items. An infit
mean square less than 0.79 indicates the item response fits the model and little distinction can be
made about the respondents based on the item. Ten items in the overall analysis have fit statistics
that lie outside the suggested range of one standard deviation above the average infit or outfit
statistic, which was 1.23 and 1.19 respectively (see Table 5). Nine out of ten of these items were
within the methodology section of the survey. Nine items overfit the model, meaning the item fit
statistics were over one standard deviation below the infit or outfit statistic which was 0.79 and
0.77 respectively (see Table 6).
[Insert Table 5 Here]
[Insert Table 6 Here]
Rating scale category function
For the survey instrument, the responses correspond to the scale where 1 = Strongly
Disagree, 2 = Disagree, 3 = Agree and 4 = Strongly Agree. The number of observations and
distribution of observations across categories were examined to describe the functioning of the
rating scale categories. The observed count indicates the number of times the category was
selected (see Table 7).
[Insert Table 7 Here]
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
22
The observed count in the category measure does increase with the rating scale as
expected. It appears from the frequencies reported that respondents are utilizing the full range of
the four-point scale. The majority of responses in the overall model lie in the agreement
categories, which indicates most survey items were likely to be endorsed by the average
participant. Advancing average measures with each category and step calibrations ensure the
rating scale measure is stable and accurate. Probability curves were used to visually inspect the
rating scale category function. The probability curves representing the function of rating scale
categories demonstrated respondents utilizing the full scale. All rating categories reached a peak
within 3 logits, which indicates respondents used all four response categories.
Empirical characteristics of responses
The item map produced from the Rasch analysis provides an empirical hierarchy for
survey items based on responses, which represents characteristics of good research that are least
to most likely to receive endorsement. The item map revealed the average person measure was
above the average item measure, which indicates the majority of the survey items were more
likely to be endorsed by participants (see Figure 1).
[Insert Figure 1 Here]
A large gap exists between survey item M2 and survey items M12, M13, M14, and M15.
The survey items 12 and 15 from the methodology section were the only two items over one
standard deviation above the average person measure. Overall, participants do not agree that
these items describe characteristics of high quality research (see Table 8).
[Insert Table 8 Here]
Summary
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
23
The overall fit of the data to the Rasch model was acceptable. The methodology
component of the survey resulted in the most misfitting items as well as the most difficult items
to endorse. Participants were distributed across all primary research methodologies, which
accounts for some discrepancies across the methodology survey items. The results provide a
foundation to guide revisions to the instrument prior to the full scale study.
Discussion
Prior to interpreting participant responses to statements concerning what constitutes good
educational research, the survey instrument itself was examined. In review of the quality of the
measure, the survey was evaluated for fit of data to the Rasch model, orientation of the latent
variable to ensure directional polarity, and functionality of the rating scale response categories.
Based on person and item reliability and separation estimates (.86 and .98; 2.48 and 7.64,
respectively), the survey instrument appears to be a reasonably reliable measure of educational
researchers' conception of what constitutes high-quality research. Point-biserial correlations
demonstrated polarity of the items in the positive direction, ensuring unidimensionality of the
survey and indicating good construct validity.
Concerning the fit of data to the one-parameter IRT Rasch model, it appears that ten
items were slightly misfitting, as indicated by infit or outfit mean square fit statistics one
standard deviation above the suggested range. Misfitting items draw survey-developers attention
to the function of these items and patterns of participant responses that do not match what is
expected. Specific to this survey, all misfitting items were methodological statements. Similarly,
nine items overfit the model, as indicated by fit statistics one standard deviation below the
suggested range. Overfitting items may provide researchers with minimal information and little
distinction between groups of participants based on such items. Items that appeared to have
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
24
overfit included all domains (ethical, methodological and theoretical). In each instance where
item data does not fit the model as intended, researchers must take a closer look at the item and
make revisions as appropriate. In the case of misfitting methodological items, however,
discrepancies in responses may be accounted for by the differences in participants' preferred
methodological approach. Another explanation may be that methodological issues are not well
articulated by the items themselves. Therefore, potentially misfitting items may need revision
rather than be taken off the survey in their entirety.
With regard to the function of the rating scale response categories, multiple analytical
approaches were taken to evaluate participant utilization of responses for items. Observed count,
frequency statistics, and probability curves demonstrate that participants utilized the full range of
the four-point scale. Although participants mostly endorsed agreement to statements about high-
quality research characteristics, each category was endorsed, ensuring that the rating scale
measure is both stable and accurate.
Person and item maps indicated which characteristics were most likely to be endorsed by
participants, thus indicating which characteristics of good educational research were least or
most salient to respondents. Overall, participants disagreed with items M12 ("High-quality
research requires random sampling") and M15 ("High-quality research can be determined solely
by examining the research methodology"). Similarly, participants showed little agreement with
items M13 ("High-quality research requires quantifiable measures of results") and M14 ("High-
quality research consists of experimental studies that yield prescriptions for actions"). Closely
related to explanations for item misfit, it is reasonable to argue that these questions were too
methodologically-based to provoke agreement by both quantitative and qualitative researchers.
On the other hand, participants generally agreed to all other items in the survey. Characteristics
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
25
with the most participant agreement included: E3 (“High-quality research should protect the
safety and welfare of participants.”), E1 (“High-quality research abides by ethical standards.”),
and M6 (“High-quality research should be methodologically sound.”).
Ethical Considerations
Faculty and student participant responses to ethics items revealed a number of opinions
reasonably aligned with the available literature regarding ethics and research. Respondents found
it less difficult to endorse compliance with unspecified ethical standards (E1), and similarly,
adherence to guidelines recognized by their professional organizations (E10). These results could
reflect an appreciation of the ethical guidelines, currently in existence, imparted by professional
associations such as the AERA and the APA. Another explanation may be that researchers in
educational disciplines generally agree with generic ethical guidelines regardless of preferred
methodological approach, but may differ according to methodology in reference to procedure-
specific characteristics of good research, as seen in responses to other, more methods-based
items.
Likewise, further results suggested that responses to items regarding recognition of
contributors to research (E6), safety and welfare of participants (E3), adherence to institutional
policies (E5), and minimal use of techniques yielding negative social consequences (E4), were
more difficult to endorse yet still more likely to evoke positive responses. These responses
reflect the content of ethical guidelines offered by professional organizations, as well as those
presented in social research texts, relating to informed consent, credit, and the use of Institutional
Review Boards.
Respondents also found it less difficult to endorse item E7, which addresses execution of
research by investigators who have completed ethics training. These results could be interpreted
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
26
as representative of knowledge of the branch of educational research literature devoted to calls
for increased ethics education. Additional items or perhaps a better description of ethics training
and education could enhance interpretation of these results.
Items E8 and E7 related to dissemination of research findings to professionals in the
field, as well as to the public were found to be more difficult for participants to endorse.
Recalling Hostetler’s (2005) argument regarding how research should serve the public’s welfare,
these findings are worthy of note. This brings into question how participants interpreted the
questions. Contrary to professional ethical guidelines regarding dissemination of information,
respondents found these items more difficult to endorse. Do faculty and students believe that
research, in general, might not always necessitate dissemination or are there specific instances in
which research should be held from other professionals or the public? Are researchers concerned
with the implications that certain results may have on practice? Refining the instrument with
additional items could lead to a greater understanding of the motives behind these responses.
In his discussion of the topic of what is “good” research, Hostetler (2005) stated, “Good
education research is a matter not only of sound procedures but also of beneficial aims and
results; our ultimate aim as researchers and educators is to serve people’s well being” (p.16).
One could suggest the findings of this study reveal the importance of ethical considerations in
research practice. When looking at the items participants found less difficult to endorse, the
majority of the items were found in the section of ethical considerations. As stated earlier, item
E3 (safety and welfare of participants) was the item that participants found easy to endorse.
Hostetler would concur with the high endorsability of this item due to researchers “respecting the
dignity and human of persons” (p.19). Because participants found items regarding ethical
practices as less difficult to endorse, a trend could be observed about the importance of ethical
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
27
practices being taught in the classroom and in the field of research.
In general, responses positively reflected the current ethical guidelines offered by
professional organizations. Modification of the survey to include items related to ethics
education would be more representative of the current body of literature, and perhaps, assist in
filling the gap between calls for more ethics education and the relationship between that
education and how it might better inform practice. Additional items regarding research
dissemination could yield a stronger understanding of how respondents interpreted these
questions, and why these items appear more difficult to endorse than others.
Methodological Considerations
An educational researcher’s choice of procedures and methodology is influenced by his
or her underlying philosophical assumptions about reality, knowledge and values. Quantitative
and qualitative approaches to research are often differentiated by the type of data collected
(numeric or descriptive), the mode of analysis (statistical or interpretive), the logic employed
(deductive or inductive) and underlying paradigms (positivism or constructivism) (Tashakkori &
Teddlie, 1998; Merriam, 1998; Creswell & Plano-Clark, 2007). Thus, the variation in participant
agreement among survey items M12 ("High-quality research requires random sampling") and
M13 ("High-quality research requires quantifiable measures of results") is reasonable as
qualitative and quantitative researchers preferred sampling procedures and types of data
collected are distinct. The research question(s) are also an influential factor contributing to a
researcher’s choice of procedures and methodology employed. A carefully constructed alignment
of research methods to research question(s) is essential if one wishes to elucidate useful and
credible findings to the research question(s) (Johnson & Onwuegbuzie, 2004; Sikes, 2004).
Results revealed that participants generally agreed with survey items M6 (“High-quality research
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
28
should be methodological sound”) and M11 (“High-quality research requires research methods
and techniques based on the nature of the research questions”). These results indicate participant
responses align well with the existing research literature on the importance of choosing sound
methods and procedures that will best address the research question(s).
Theoretical Considerations
When viewing the findings from the section on theoretical considerations, there are
multiple connections between the findings and the literature. With regard to item fit, the analysis
revealed six items in this section that were overfitting ( T1, T7, T10, T11, & T12). The
overfitting of these items suggests that they can be used to measure the construct, although there
is not much of a distinction between the items. Items T10 (“High-quality research should be
useful to other professionals), T11 (“High-quality research connects the work to its impact on
human-well being”, and T12 (“High-quality research merges reason and value) appear to attest to
the expansion of knowledge within and across disciplines. As presented in the literature review,
theory can be viewed as an action guide in research applied to predict certain outcomes.
Participants found items T2 (“High-quality research should provide the rationale for the
conceptual orientation of the study”) and T3 (“High-quality research should provide the rationale
for the theoretical orientation of the study”) as less difficult to endorse which supports the notion
of implementing theory as the guiding factor of high-quality research. Also, participants
identified items T4 (“Educational researchers should adhere to the standards of their own
theoretical perspectives to achieve high-quality research”), T5 (“Reliability, validity, and
trustworthiness are the most important considerations in high-quality research”), and T6 (“High-
quality research provides objective answers to research questions”) as items more difficult to
endorse. The higher level of difficulty could be the result of items having a strong correlation to
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
29
professional ethics (T4) or methodological practices (T5 and T6). Overall, results indicate that
the participants appear to be cognizant of the importance of theory in high-quality research.
Conclusion
While it seemed reasonable to seek consensus among the community of education
researchers of what constitutes good research, evidence from this study supports that variability
exists across the general themes of standards in conducting research in education. This study
aimed to operationalize good education research as it pertains to three major domains (Ethics,
Methods, and Theory) by using AERA and APA as guiding authorities. The primary expectation
was to reveal which characteristics are most frequently endorsed by members of various
disciplines within educational research through examining relative frequency tables as well as
the empirical hierarchy of survey items illustrating item endorsability. The secondary purpose
was to investigate and illustrate the fit of the data to a Rasch measurement model to support the
quality of the survey measure employed and to guide revisions prior to the full scale study.
While a consensus was not determined as to what constitutes good research, attention
should be given to the characteristics that did emerge as highly endorsed items as well as items
that participants were less likely to endorse. All the items within the ethics and theory domains
were empirically found to be relatively easy to endorse for the majority of participants; however,
more variation existed in terms of item endorsability within the methods domain. Survey items
M12, M13, M14, and M15 were the most difficult items for participants to endorse. Of these,
items M12 and M13 were also highlighted as problematic items in terms of fit to the model and
will be further reviewed.
As revisions to the survey instrument are made, further study is needed to determine
variations in what constitutes good research in colleges of education across the nation. This study
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
30
stimulates dialogue how each characteristic functions as a driving force for education research,
as well as whether each should be a foundation for the research that educational researchers
conduct. Results will further challenge researchers to be mindful of characteristics of their own
research, which will ultimately drive better research in education across disciplines.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
31
References
American Educational Research Association (2000). Ethical Standards of the American
Educational Research Association: Retrieved August 15, 2007, from
http://www.aera.net/uploadedFiles/About_AERA/Ethical_Standards/EthicalStandards.
American Psychological Association (2002). Ethical Principles of Psychologists and Code
of Conduct. Retrieved August 15, 2007, from http://www.apa.org/ethics/code2002.pdf
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43,
561-574.
Babbie, E. (2008). The basics of social research. Belmont, CA: Thomson Wadsworth.
Bond, T., & Fox, C. (2001). Applying the Rasch model: Fundamental measurement in the human
sciences. Mahwah, NJ: Lawrence Erlbaum Associates.
Brandon, D. (2000). What is research? Retrieved August 15, 2007, from
http://www.scitech.technion.ac.il/faq/research.html
Bridges, D. (2001).The ethics of outsider research. Journal of Philosophy of Education. 35, 371-
386.
Bruhn, J., Zajac, G., Al-Kazemi, A., & Prescott, Jr., L. (2002). Moral positions and academic
conduct: Parameters of tolerance for ethics failure. The Journal of Higher Education, 73,
461-493.
Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd Ed.). Thousand Oaks, CA: SAGE Publications, Inc.
Creswell, J., & Plano-Clark, V. L. (2007). Designing and conducting mixed methods research.
Thousand Oaks, CA: SAGE Publications, Inc.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
32
Creswell, J. W., Plano-Clark, V. L., Gutmann, M., & Hanson, W. E. (2002). Advanced
mixed methods research designs. In A. Tashakkori, & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209-240).
Thousand Oaks, CA: Sage.
Ebbs, C. (1996). Qualitative research inquiry: Issues of power and ethics. Education. 117,
217-223.
Elmes, D. G., Kantowitz, B. H., & Roediger III, H. L. (1996). Research Methods in Psychology,
Fifth Edition. New York: West Publishing Company.
Hostetler, K. (2005). What is “good” education research? Educational Researcher, 34(6), 16-21.
Howe, K., & Eisenhart, M. (1990). Standards for qualitative (and quantitative) research: A
prolegomenon. Educational Researcher, 19(4), 2-9.
Hughes, W. P. (1999). Constructions research: A field of application. Australian Institute of
Building Papers, 9, 51-58.
Iutcovich, J., Kennedy, J., & Levine, F. (2003). Establishing an ethical climate in support
of research integrity: Efforts and activities of the American Sociological
Association. Science and Engineering Ethics, 9, 201-205.
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm
whose time has come. Educational Research, 33(7), 14-26.
Kansanen, P. (2003). Pedagogical ethics in educational research. Educational Research and
Evaluation. 9, 9-23.
Kerse, N. & Elley, R. (2003). What constitutes good research in general practice? The New
Zealand Family Physician, 30 (6), 385-387.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
33
Linacre, J. M. (2004). Optimizing rating scale category effectiveness. In E. V. Smith, Jr. and R.
M. Smith Introduction to Rasch Measurement (pp. 258-278) Maple Grove, MN: JAM
Press.
Linacre, J. M. (2005). WINSTEPS Rasch measurement computer program. Chicago:
Winsteps.com.
Merriam, S.B. (1998). Qualitative research and case study applications in research. San
Francisco, CA: Jossey-Bass.
Muskavitch, K. (2005).Cases and goals for ethics education. Science and Engineering
Ethics, 11, 431-434.
Nardi, P. (2003). Doing survey research: A guide to quantitative methods. Boston, MA:
Pearson Education, Inc.
Simmerling, M., Schwegler, B., Sieber, J., & Lindgren, J. (2007). Introducing a new
paradigm for ethical research in the social, behavioral, and biomedical sciences:
Part I. Northwestern University Law Review, 101, 837-859.
Sikes, P. (2004). Methodology, procedures and ethical concerns. In C. Opie (Ed.), Doing
education research (pp. 15-33). Thousand Oaks, CA: SAGE Publications, Inc.
SurveyMonkey [Survey software program] (2007). Copyright 1999-2007 SurveyMonkey.com.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and
quantitative approaches. Thousand Oaks, CA: SAGE Publications, Inc.
Whitley Jr, B. E. (1996). Principles of Research in Behavioral Science. Mountain View, CA:
Mayfield Publishing Company.
Wright, B. D., & Masters, G. N. (1982). Rating scale analysis: Rasch measurement. Chicago, IL:
MESA Press.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
34
Wright, B. D., & Stone, M. H. (2004). Making Measures. Chicago, IL: The Phaneron Press.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
35
Table 1 Personal Demographic Characteristics of Faculty Respondents (n = 50) Characteristic n % Faculty Rank
Instructor 3 5.9 Lecturer 0 0 Assistant Professor 15 29.4 Associate Professor 12 23.5 Professor 13 25.5 Visiting Professor 2 3.9 Adjunct Professor 1 2.0 Part-time Professor/Instructor 2 3.9 Other 3 5.9
Years as a faculty member (at any higher education institution)
Less than 1 year 6 12.0 1-3 years 12 24.0 4-6 5 10.0 7-9 3 6.0 10-15 7 14.0 16-20 4 8.0 Greater than 20 years 13 26.0
Tenure Status
Tenured 27 52.9 On tenure track, but not tenured 13 25.5 Not on tenure track 11 21.6
Age 71 or older 2 4.3 61-70 6 12.8 51-60 12 25.5 41-50 7 14.9 31-40 15 31.9 Less than 30 5 10.6
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
36
Table 2 Personal Demographic Characteristics of Graduate Student Respondents (n = 76) Characteristic n % Status Graduate Assistant 3 4.1
Research Assistant 19 25.7 Teaching Assistant 14 18.9 Doctoral Student 39 52.7 Master’s Student 8 10.8 Specialist Degree Student 8 10.8 Other 12 16.2
Years as a graduate student (at any institution)
Less than a year 8 11.0 Between 1-2 years 14 19.2 Between 3-4 years 21 28.8 Between 5-6 years 18 24.7 7 or more years 12 16.4
Age Under 20 0 0
20-24 12 16.0 25-29 23 30.7 30-34 10 13.3 35-39 10 13.3 40-44 5 6.7 45-49 5 6.7 50-54 6 8.0 55 or older 4 5.3
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
37
Table 3 Primary Academic Interest for Faculty and Graduate Student Respondents (n = 121) Educational area n % Administration, Organization & Leadership 6 5.0 Curriculum Studies 3 2.5 Learning & Instruction 21 17.4 Measurement & Research 8 6.6 Counseling & Human Development 14 11.6 History & Historiography 2 1.7 Social Context of Education 10 8.3 School Evaluation & Program Development 5 4.1 Education in the Professions 3 2.5 Postsecondary Education 7 5.8 Teaching & Teacher Education 17 14.1 Educational Policy & Politics 6 5.0 Other 19 15.7
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
38
Table 4 Interests of Faculty and Graduate Student Respondents (n = 76) Item n % Primary interest in teaching or research
Very heavily in research 11 9.2 In both, but leaning toward research 41 34.2 In both, but leaning toward teaching 57 47.5 Very heavily in teaching 11 9.2
Hours spent on research each week
None 13 10.5 1-9 57 46.0 10-19 30 24.2 20-29 14 11.3 30-39 3 2.4 40 or more 7 5.7
Primary research methodology
Entirely qualitative 4 3.4 Mostly qualitative 25 21.2 Mixed methods (50/50) 41 34.8 Mostly quantitative 31 26.3 Entirely quantitative 17 14.4
Perception of Educational Research Quality
Very low quality 1 0.8 Low quality 14 11.7 Average quality 69 57.5 High quality 35 29.2 Very high quality 1 0.8
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
39
Table 5
Item fit statistics poorly fitting the model Item Number Count Measure Infit Outfit
M10 136 0.86 1.38 1.40
M12 136 2.91 1.40 1.40
M3 137 -0.26 1.36 1.30
M6 137 -1.77 1.31 1.07
M2 135 1.39 1.18 1.31
M17 133 0.74 1.27 1.30
M4 137 -1.33 1.25 1.07
M13 136 2.43 1.25 1.25
M1 136 -0.54 1.25 1.18
E4 127 -0.16 1.20 1.23
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
40
Table 6
Item fit statistics overfitting the model Item Number Count Measure Infit Outfit
T1 126 -0.47 0.78 0.83
T11 127 0.32 0.80 0.78
M16 136 -0.35 0.78 0.76
T7 125 0.39 0.73 0.73
T10 127 -0.42 0.73 0.69
E9 127 0.33 0.71 0.72
E10 129 -1.01 0.67 0.69
T9 127 -0.44 0.68 0.66
T12 123 0.35 0.63 0.65
Table 7
Summary of Rating Scale Diagnostics Category Observed INFIT OUTFIT
Count (%) Mean Square Mean Square Strongly Disagree 175 (3%) 1.58 1.77 Disagree 715 (13%) 0.87 0.82 Agree 2167 (41%) 0.94 0.87 Strongly Agree 2042 (38%) 0.99 0.99
Note. Category, observed count, and percentage indicate the numbers of respondents who chose a particular response category, summed for each category across all 39 items.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
41
Figure 1. Map of survey items by difficulty to endorse
<more>|<difficult to endorse> | | 4 . + # | # T| | # | 3 .## + M12 M15 .## | ### S|T #### | M13 M14 .## | 2 ###### + ###### M| .####### | .###### | M2 ######## |S T8 1 .####### + T6 .## S| M10 M17 M5 ### | T4 T5 ## | E9 T11 T12 T7 . | E8 0 T+M . | E4 E7 M3 T3 | M16 T1 T10 T9 | E5 M1 T2 | E6 M11 -1 + E10 |S E2 M7 M8 | M4 M9 | | M6 -2 + E1 | E3 | |T | -3 + <less>|<difficult to endorse> EACH '#' IS 2.
Perceptions of Education Research
*** Working Draft *** Do not cite without permission of contact author.
42
Table 8
Counts and percents of responses for highlighted survey items No. 1 2 3 4 Survey Item Description
M12 37 (27%) 70 (51%) 22 (16%) 7 (5%) High-quality research requires random sampling.
M13 27 (20%) 60 (44%) 42 (31%) 7 (5%) High-quality research requires quantifiable measures of results.
M14 24 (18%) 63 (47%) 41 (30%) 7 (5%) High-quality research consists of experimental studies that yield prescriptions for actions.
M15 29 (21%) 92 (68%) 13 (10%) 2 (1%) High-quality research can be determined solely by examining the research methodology.