Research methods for graduate students: a practical framework to guide teachers and learners

EDUCATION

Research methods for graduate students: A practical frameworkto guide teachers and learnersPatricia F. Pearce,MPH, PhD, FNP, FAANP (Associate Professor)1 , Becky J. Christian, PhD, RN (Professor andInterim Chair, Family, Child Health, & Caregiving Department School of Nursing)2, Sandra L. Smith, PhD, APRN,NNP-BC (Associate Professor)3, & David E. Vance, PhD, MGG (Associate Professor, Associate Director, Center forNursing Research)2

1Loyola University College of Social Sciences, School of Nursing, New Orleans, Louisiana2University of Alabama at Birmingham School of Nursing, Birmingham, Alabama3University of Louisville School of Nursing, Louisville, Kentucky

KeywordsResearch; education; critical thinking;

pedagogy; evidence-based practice;

curriculum.

CorrespondencePatricia F. Pearce, MPH, PhD, FNP, FAANP,

Loyola University, College of Social Sciences

School of Nursing, 6363 St. Charles Avenue,

Stallings Hall #212C, New Orleans, LA 70118.

Tel: 205-541-8988; E-mail: [email protected],

[email protected]

Received: June 2013;

accepted: August 2013

doi: 10.1002/2327-6924.12080

Abstract

Purpose: The purpose of this article is to present the Arrow Framework for Re-search Design, an organizing framework that facilitates teaching and learning ofresearch methods, providing logical organization of interrelationships betweenconcepts, content, and context of research methods, and practice application.The Arrow Framework was designed for teaching and learning research methodsto facilitate progression of knowledge acquisition through synthesis.Data sources: The framework was developed over several years and usedsuccessfully to teach masters, DNP, and PhD nursing students across five uni-versities. The framework is presented with incremental graphics and narrativefor teaching.Conclusion: The Arrow Framework provides user-friendly information, in anorganized and systematic approach demonstrated as successful for teachingand learning the foundational language of research, facilitating synthesis andapplication in scholarly endeavors.Implications for practice: The Arrow Framework will be useful for educa-tors and students in teaching and learning research language, relationships,and application of methods. The materials are easily adaptable to slide or pa-per presentation, and meet learner needs for narrative and visual presentation.Teaching research design to graduate students is critical to meet the expecta-tion that students are to understand the scientific underpinnings of nursingscience and appropriate use of evidence that are essential for well-educatedpractitioners.

Introduction

Understanding the scientific underpinnings of nursingis a critical skill for all nurses, and is the hallmark of anursing professional, as well as a major component of theEssentials for the master’s degree in nursing (MSN),the clinical doctor of nursing practice (DNP) programs,and recommendations for doctor of philosophy (PhD)programs (American Association of Colleges of Nursing[AACN], 2001, 2006, 2011). Expectations for nurses,such as those identified by the Institute of Medicine(2010), to lead change in the quest for advancing healthcare require an understanding of evidence, as well as

the generation and practical use of evidence upon whichto base practice. These themes are echoed in a recentcall to action for nursing (Benner, 2010), and in nursingresearch and evidence-based practice textbooks (Grove,Burns, & Gray, 2013; Hall & Roussel, 2014; Melnyk &Fineout-Overholt, 2011; Polit & Beck, 2012).

Regardless of program emphasis, a critical factor tounderstanding science lies in understanding an encom-passing perspective of evidence, including the language ofresearch, research methods, and the use of data. Com-pletion of any graduate program and related scholarlyprojects encompasses use of all levels of Bloom’s taxon-omy that hierarchically builds upon the acquisition of

1Journal of the American Association of Nurse Practitioners 00 (2013) 1–13 C©2013 The Author(s)C©2013 American Association of Nurse Practitioners

Research methods for graduate students P. F. Pearce et al.

knowledge, then comprehension, application, analysis,synthesis, and evaluation (Anderson & Krathwohl, 2001;Bloom, 1974; Forehand, 2005). The emphasis in theclinically oriented MSN and DNP programs is to prepareadvanced practice nurses (APRN) and highly skilled clin-icians, who must be able to discuss evidence adeptly tofunction fully in the rapidly evolving healthcare arena.MSN- and DNP-prepared nurses, including APRNs, mustbe able to critically appraise and critique published re-ports that include evidence, and then translate that evi-dence into practice and practice-related scholarly projects(AACN, 2006, 2011). Further, for APRNs, understandingof evidence is required in order to evaluate their prac-tice outcomes and productivity. Thus, to understand theprocess, application, and translation of science, graduatenursing education necessarily must include courseworkfor MSN and DNP students that is rich in research meth-ods, basic statistics, and the overall use of evidence, butnot as in-depth as required in a PhD program (AACN,2006, 2011).

Purpose

The purpose of this article is to present the Arrow Frame-

work for Research Design, an organizing framework that fa-cilitates teaching and learning of research methods, withlogical and practical organization of the interrelation-ships between concepts, content, and context of researchmethods, as well as application of evidence to practice.The Arrow Framework was developed from several yearsof experience in teaching research methods and statis-tics, and then used effectively by four faculty at five uni-versities, in teaching hundreds of MSN, DNP, and PhDstudents. The practical framework provides both educa-tors and students with an overall schematic and visualdisplays for organizing research methods content, under-standing the hierarchy of designs and level of evidence,facilitating the educator’s ability to present the related in-formation on the research designs within the paradigmsin an understandable manner, and providing ample op-portunities for faculty to address the paradigmatic philo-sophical and historical underpinnings. Further, use of theArrow Framework enhances the student’s ability to graspsufficient research knowledge to critically evaluate re-search reports, develop their scholarly projects, and usethe information adeptly in practice. The framework isintroduced in a layered, hierarchical, systematic format,and presented sequentially in a visual format to studentsto facilitate critical thinking.

Application of the framework to the major researchparadigms, quantitative, qualitative, and mixed methods,and their related components is addressed, and exam-ples are included to illustrate the concepts. The quanti-

tative paradigm (positivist), the qualitative paradigm (alsocalled naturalistic, interpretive, or constructivist), and themixed methods paradigm (pragmatic) are well-established(Creswell, 2012, 2013; Denzin & Lincoln, 2011; Polit &Beck, 2012). The philosophical foundations are empha-sized in the framework using only categorical descriptors.Augmenting class discussions about the foundations ofresearch and evidence-based practice is critical for studentunderstanding and helpful for students to understand theoverall paradigms and designs.

Background and significance

The use of evidence, therefore necessarily research, is thefoundation of nursing practice, established by FlorenceNightingale in the late 1800s (Dossey, 2010). Educatingnurses in the language and practice of research, how-ever, is challenging and many barriers exist to integrat-ing research into practice (DiCenso, Guyatt, & Ciliska,2005; Fineout-Overholt, Melnyk, & Schultz, 2005; Funk,Tornquist, & Champagne, 1995; Hall & Roussel, 2014;Jones, Crookes, & Johnson, 2011; Melnyk & Fineout-Overholt, 2011; Melnyk et al., 2004; Polit & Beck, 2012;Schmidt & Brown, 2011). Understanding research lan-guage is one barrier that has been well documented inthe literature (Kelly, Turner, Gabel Speroni, McLaugh-lin, & Guzzetta, 2013; McLaughlin, Gabel Speroni, Kelly,Guzzetta, & Desale, 2013). In a systematic review of 45published reports on research utilization, Squires, Es-tabrooks, Gustavsson, and Wallin (2011) demonstratedthat a positive attitude of nurses to research was the pri-mary factor related to research utilization. APRN iden-tified similar barriers to using evidence in practice; yet,APRNs are licensed to provide advanced care and, there-fore, they must be highly skilled at evaluating evidence ata more in-depth level, applying evidence in practice, andgenerating practice-based initiatives that rely on evidenceand outcomes. In a descriptive qualitative study of transi-tion from RN to APRN role, understanding evidence andits use in practice was identified as by participants as ahigh priority (Spoelstra & Robbins, 2010).

Data are considered to be evidence and nurses use dataand evidence in every patient encounter. For example,measurement of laboratory values provides data that areessential for everyday practice. Each value is scrutinizedfor the expected range in relationship with patient ageand gender. Critical decision making is necessary regard-ing the validity of the measurement, as well as applicationin the contextual issues related to the patient situation.However, laboratory values are not generally consideredby nurses and students to be a form of data or evidence.Further, nurses and students have difficulty translating

2

P. F. Pearce et al. Research methods for graduate students

the use of these data and evidence from clinical practiceto the language and context of research.

Similarly, differential diagnosis requires an extensiveunderstanding of data and evidence from history andphysical exam to determine a culminating diagnosis, butthese related data are not labeled necessarily as evidenceby APRNs. Although data in clinical practice are consid-ered clinical data, or clinical evidence, and are not collectedgenerally for research purposes, clinical examples help totranslate research concepts and principles for clinicians.Introducing research language, such as dependent andindependent variables, validity and reliability, level of evidence,and causality, typically produces glazed looks or silencefrom students. By providing the linkages to clinical prac-tice, the concepts become more clear, and the applicationof information is more easily understood, and studentsare better able to generalize the clinical concepts to re-search. Understanding the basic concepts and definitionsrelated to research is essential for developing foundationalknowledge about evidence, and moving toward criticalevaluation and application of the more in-depth detailsof research.

Every graduate program includes one or more researchcourses with labels ranging from scholarly inquiry, evidence-based practice, research, or research methods, and with somethat include statistical content and others with statis-tics as a separate course. Information in these coursesis scaled to the level of the educational program, withmore in-depth information and expectations for learn-ing, progressing from BSN, through MSN, DNP, and PhDprograms. Often research is taught in graduate nursingprograms as a survey course, yet the content material isdense and difficult to cover thoroughly without creatingconfusion. Students are expected to move from knowl-edge acquisition through application and synthesis in onesemester or through several sequential courses. However,the baseline for all levels requires students to understandresearch language, such as research design, analysis, critique,and level of evidence (AACN, 2006, 2011; Benner, 2010;Polit & Beck, 2012).

The Arrow Framework for research design

In order to illustrate the range of research designs, aswell as distinct differences in research paradigms and therelationships of the designs to each other, a graphic inthe form of an arrow was created as the visual frame-work. The arrow serves as the foundation for represent-ing research paradigms, and then superimposing addi-tional information, building the detail and complexity ofresearch methods. A visual graphical display is a usefultechnique for presenting information in a manner thatsupports both teaching and learning a new language, cre-

ating a framework to make complex information moreorganized, providing logical linkages between concepts,and addressing information from knowledge acquisitionthrough synthesis (Forehand, 2005; Harris, 1996; Tufte,1990, 2001).

The foundational arrow depicting the relationship be-tween qualitative, quantitative, and mixed methods researchparadigms (see Figure 1) then is augmented with exten-sive, incremental layering of related research methods,scientific rigor, and analysis details. The arrow was se-lected as the visual form for presenting research meth-ods framework because the arrow shows the continuum orrange, without prioritizing one facet over another, as rec-ommended for high-quality symbol use in graphic design(Harris, 1996; Tufte, 1990, 2001). Thus, the arrow visu-ally demonstrates the continuum of research paradigms.As additional concepts and more complexity are intro-duced, the arrow is expanded to include multiple layerswith additional concepts superimposed to depict these in-terrelationships. With the introduction of each new facetof research, the original information moves to the back-ground (gray), and new information highlighted in bold.Thus, the basic arrow transforms into a series of arrowsused throughout the course to illustrate and translate theincreasing complexity of research designs.

Major research paradigms

With a need for educators to explain the researchcontinuum, and students to understand the predomi-nant traditions in research, commentary accompanies theArrow Framework (see Figure 1) to explain the range ofresearch, including the philosophical foundations of theresearch paradigms. The left side of the arrow representsthe qualitative design paradigm, the right side representsthe quantitative design paradigm, and the center depictsmixed methods. To support students’ understanding of thecontinuum and range, an analogy to the differences be-tween political parties ranging from conservative and lib-eral political viewpoints found in many countries hasdemonstrated usefulness, and generally is easily under-stood. Thus, using the horizontal axis, the right side of thearrow is perceived as the quantitative research paradigmthat is more conservative, traditional, and corresponds todeductive scientific method that students learned in grade-school science; while the left side is depicted as the qual-itative research paradigm that is less traditional, less conser-

vative, more liberal viewpoints, predominantly using moreinductive processes than deductive; and the middle section,mixed methods, represents a more moderate stance that in-cludes components of designs from each of the anchor-ing ends, or paradigms. On the vertical axis, extremes fallat the outer ends of the arrow, and the baseline, with

3


Major Research ParadigmsResearch Designs

MIXED METHODS

QUANTITATIVEQUALITATIVE

Figure 1 Three major research paradigms.

the least complex at the inner lower aspects of the arrow.The philosophical and historical nuances of the paradigmscan be introduced with this initial arrow in orderthat students understand the basis for each paradigm. Itis important to note that the characteristics of the vari-ous paradigms are not necessarily right or wrong, or betteror worse from a research methods perspective, but sim-ply represent differing approaches along a continuum ofresearch.

Research designs

Once the idea of a continuum with the major researchparadigms is introduced, the next step involves superim-posing the associated research design labels with the re-spective paradigms on the arrow (see Figure 2). Criticalto this presentation is explaining the stair-stepped layer-ing pattern comprising each paradigm that is representedin upward (vertical) and outward (horizontal) placementof the terms. The easiest manner of explaining the Ar-row Framework is with the stair-step design pattern; theincreasing design complexity is depicted with movement ofthe designs outward along the continuum . The most ba-sic design in each paradigm begins with descriptive design,moving vertically upward and horizontally outward (tothe right for quantitative, and left for qualitative) on thearrow to the more complex design, representing interpre-tive orientation. A caveat is that the hierarchical nature ofquantitative design is clear and generally accepted; how-ever, for the qualitative paradigm, the hierarchy of de-signs is dependent on the complexity of the analysis andoverall procedures (Creswell & Plano Clark, 2011; Denzin& Lincoln, 2011; Morse & Field, 1995; Morse & Niehaus,2009; Tashakkori & Teddlie, 2003a). Thus, a bracket tothe left in the qualitative design area of the Arrow Frame-

work indicates the level of complexity in these designs,especially as related to interpretive analysis, can vary;thus, the designs do not fall into a clear hierarchical fash-ion. For mixed methods, the complexity is determined bythe types of design combined from the quantitative andqualitative paradigms, the emphasis on the type of de-

sign, and the sequencing of the related activities (Creswell& Plano-Clark, 2006; Denzin & Lincoln, 2011; Morse& Field, 1995; Morse & Niehaus, 2009; Tashakkori &Teddlie, 2003a). The listing of designs provides the stu-dent with information regarding generally accepted levelof evidence, ranging from the lowest to the highest levelswithin the qualitative and quantitative designs, as well asacross the research paradigms. Because mixed methodsdesign is understood most easily in context of the designsselected from the quantitative and qualitative paradigms,generally presentation of the details of mixed methodsdesign follows discussion of quantitative and qualitativedesigns.

However, students must be cautioned that the researchquestion posed by the investigator drives study design

choice, and design choice subsequently influences level ofdata and overall evidence. A qualitative design is con-sidered as high level of evidence for research questionsaimed at understanding the meaning of a phenomenon;whereas, a quantitative design is more frequently usedfor comparative questions (Polit & Beck, 2012). The ma-jor research designs are presented in the framework(Figure 2), and although this list is extensive, it is notexhaustive and other designs may be included at the dis-cretion of the faculty. The designs and related level of evi-dence produced are extensively detailed in research meth-ods textbooks (Grove et al., 2013; Polit & Beck, 2012).

Research question and hypothesis

With readings and discussion, students generally graspa basic understanding of the research designs andparadigms, but with introduction of research questions andhypotheses, students can become confused. Explanation ofthe differences and uses of research questions and hy-potheses is important, because research questions andhypotheses should link directly to the research designchoice, and this relationship to design is shown in theArrow Framework graphic (Figure 3). Superimposing theidea of research question and hypothesis, this arrow coversthe entire paradigmatic span, in which a research question

4


QUANTITATIVEQUALITATIVE

Major Research Paradigms

Meta-AnalysisMulti-Site RCT

Interventional/RCTExperimental

Quasi-ExperimentalCorrelational

Cross-SectionalCase Control

Descriptive

Qualitative and Quantitative DesignsMeta-SynthesisMeta-Summary

Discourse AnalysisPhenomenology

Grounded TheoryEthnography

NarrativeInterpretive

Descriptive

MM

Figure 2 Major research designs by paradigm, emphasizing qualitative and quantitative.

Note. MM,mixedmethods; RCT, randomized controlled trial. For qualitative research designs (bracket on left) between descriptive (baseline, foundational

design) and metasummary and metasynthesis that require inclusion of multiple already existent research, the designs can be layered in a variety

of different orders, representing complexity in interpretation. There is no one hierarchical schema for qualitative (constructivist) design. The Arrow

Framework reflects overall designs and a general framework orientation.



RCTExperimental



Descriptive

Research Designs

Meta-SynthesisMeta-SummaryDiscourse AnalysisPhenomenology

Grounded TheoryEthnographyNarrative

InterpretiveDescriptive

Research Question and Hypothesis

Research Question

Hypothesis

QUANTITATIVEQUALITATIVE MM

Figure 3 Research questions and hypotheses across paradigms.

Note. MM, mixed method; RCT, randomized control trial.

can be used appropriately, and designates a formal hypoth-

esis as used only in quantitative design when sufficientevidence exists to formulate a hypothesis (Figure 3).

Sampling design

The next layer of design characteristics focuses on sam-pling design, a critical factor for any research or project de-sign (Grove et al., 2013; Polit & Beck, 2012). Definitionsand application of sampling design must be discussed, butpositioning within the arrows helps students to solidifytheir thoughts about the relationship of sampling designto the respective research paradigms, and the research

designs within the paradigms. Figure 4 includes desig-nation of the predominant form of sampling with pur-posive sampling, unique to qualitative research, and con-venience sampling in quantitative design. Probability sam-

pling aligns with the more complex quantitative researchdesigns and rarely, if ever, occurs in qualitative designs.Students are often confused by sampling designs reportedin published literature, thus, a critical piece of informa-tion in explaining sampling design is that researchers of-ten use a creative mix of various forms of sampling, in-cluding strata, staged, quota, etc. (Polit & Beck, 2012). Inmixed methods design, multiple forms are also used. Thetendency in sampling design decisions is for heightened

5




RCTExperimental



Descriptive

Research Designs




Sampling Design

Purposive Convenience


Probability

Figure 4 Sampling design across research designs and paradigms.


representativeness in the quantitative designs and for moreemergent sampling in qualitative designs. However, it is es-sential for students to understand how researchers mixsampling designs, and how to identify integrated sam-pling designs. The arrow schema provides an overall de-piction, with the details instructed in discussions of sam-pling designs overall.

Data types and level of measurement

Data types and level of measurement are introduced aftersampling design in the Arrow Framework (Figure 5). Eachresearch paradigm is hallmarked by the specific type ofdata that comprise the predominant focus. In qualitativeresearch, investigators emphasize narrative text, while nu-meric orientation is emphasized in quantitative research,and both forms are used in mixed methods design. Thedistinction between qualitative and quantitative designsis not to imply that qualitative researchers ignore num-bers, counting, or other numeric facets of their research,but to emphasize the preponderant orientation to data ineach paradigm (Chang, Voils, Sandelowski, Hasselblad, &Crandell, 2009; Morse & Field, 1995; Polit & Beck, 2012;Sandelowski, 2001, 2010).

It is important to note that the complexity of interpreta-tion of both narrative and numeric data increases as thedesigns become more complex, that is, with movementupward and outward in the designs listed on the arrow,the more complex the level of data and interpretation,and the more removed from the raw data, regardless ofqualitative or quantitative research designs. For example,in qualitative design, metasynthesis requires inclusion of allpublished qualitative research reports on the given topic,

thus, is more complex and highly interpretive than de-scriptive design studies, and is placed farther out and up-ward on the arrow. In contrast, with descriptive qualita-tive research, analysis stays in close relationship to theraw data (Sandelowski, 2000b, 2010). Designs betweendescriptive and the metasummary and metasynthesis de-signs in the qualitative paradigm are reordered to repre-sent increasing complexity in analysis and interpretation.In the quantitative paradigm, meta-analysis is consideredthe highest form of evidence (Polit & Beck, 2012); how-ever, meta-analysis is a form of secondary analysis that isremoved from the raw data analysis in the primary re-search studies, because the data are often drawn frompublished studies, or raw data are frequently obtainedfrom researchers for the secondary analysis.

Scientific rigor

Overall scientific rigor, required in the paradigms, whileconceptually has similar application across the researchparadigms, varies in philosophical basis, function, and ap-plication, the components of scientific rigor are particu-larly difficult for students to understand (Figure 6). Thedifficulty is especially true regarding systematic data col-lection and analysis of quantitative data, and the psycho-metric properties of reliability and validity of measurementtools. The use of the terms reliability and validity in rela-tionship to properties of a data collection instrument, orthe manner in which measurements are performed, arefrequently confused by students with the terms internal

and external validity in relationship to the reported study.It is critical for students to understand contributions tointernal validity from conceptualization, identification of

6




RCTExperimental



Descriptive

Research Designs




Data TypesNarrative Numeric


Figure 5 Data types for each paradigm: narrative and numeric.




RCTExperimental



Descriptive

Research Designs




RigorNarrative Numeric


Measures--Reliability and ValidityInternal ValidityExternal Validity

TrustworthinessCredibilityConsistencyTransferability

ConfirmabilityCreativityCongruenceCriticalityExplicitnessThoroughnessSensitivityVividness

Figure 6 Scientific rigor.


study variables, sampling design, and data collection pro-cedures and analysis. External validity, or generalizabil-ity, entails the larger picture of understanding of whetheror not the research results can be placed into contextof the larger target population or similar populations,geographic location and settings, and time. The under-lying constructs, definitions, and operationalization ofinternal and external validity in relation to scientific rigorare essential for the design of a quantitative researchstudy, and for understanding research and for evaluatinga published report. For example, it is critical for students

to understand that investigators who conduct quantita-tive research emphasize the importance of external va-lidity; while the potential for applying findings to otherpopulations is emphasized less in qualitative design, but isconsidered as one aspect of transferability in the qualitativeparadigm.

In the quantitative research paradigm, reliability and va-

lidity of data collection measures or tools are paramount,including: (a) varying forms of reliability (e.g., internalconsistency, test–retest, parallel form), (b) validity (e.g., face,content, construct, convergent, discriminant), (c) application

7


for data collection processes (inter-rater and intraraterreliability), as well as the overall notions of internal

validity and external validity (Grove et al., 2013; Polit &Beck, 2012). Each of these procedures requires detailedinformation for students; however, the Arrow Framework

provides a context for the forms of rigor used. A vitalcomponent for student understanding is the linkagebetween the level of data, measurement parameters andprocesses, and level of evidence.

Rigor in qualitative design includes emphasis on cred-

ibility, trustworthiness, overall consistency, and transferabilityand numerous techniques are used to assure that rigoris met (Denzin & Lincoln, 2011; Morse, 1999; Morse,Barrett, Mayan, Olson, & Spiers, 2002; Sandelowski,1986, 1993). The basic components expand further toinclude components of criticality, congruence, explicitness,integrity, sensitivity, and vividness (Whittemore, Chase, &Mandle, 2001). These components of scientific rigorare superimposed on the arrow graphic in relationshipto the qualitative and quantitative research paradigms(Figure 6), and are used selectively in mixed meth-ods design. Thus, rigor is desired and useful for eachof the major paradigms, but viewed in a differentmanner, and established with differing techniques andprocedures.

Data analysis

Figure 7 represents information needed for the twomajor steps of data analysis, crossing the spectrum ofresearch paradigms and research designs in the ArrowFramework. Although in most studies there are multiplesteps in data analysis, those more comprehensive stepsare simplified for presentation purpose, and discussedwith more detail in course instruction. In Step 1, descrip-tive statistics, such as frequencies for categorical variables(nominal and ordinal), and measures of central tendencyand variability for continuous variables (interval and ratiodata) are used by both qualitative and quantitative re-searchers. At minimum, investigators use these statisti-cal calculations for describing sample characteristics andoverall demographics.

Although investigators working in qualitative researchemphasize primarily textual, narrative data, counting ortallying in terms of codes or related study artifacts is im-portant. In quantitative research, in addition to the sam-ple demographics, results for all variables are includedin the descriptive statistics step analysis (Step 1). It iscritically important to emphasize to students that basicdescriptive statistics (e.g., frequency, measures of central ten-dency, and variability) are used in describing the character-istics of the sample in both qualitative and quantitative

design paradigms in order to understand study context,as well as applicability (Figure 7).

Data analysis Step 2 (see Figure 7) varies substantiallyacross the paradigms and research designs. In qualitativeresearch, data analysis emphasizes words or narrative text,thus, words are the data that are managed and analyzed,and are the primary unit of analysis. Qualitative analy-sis procedures generally align with the methodologic tra-dition, but remain flexible. In general, content analysisbegins with the identification of categories, themes, andpatterns for basic descriptive analysis of data. The Ar-row Framework includes the major traditions, with base-line descriptive orientation at the lowermost portion ofthe arrow with metasynthesis at the uppermost portionvertically.

For quantitative research design (Figure 7), numeric-based analysis ranges from simple descriptive statistics,through inferential (parametric and nonparametric; e.g., chi-square, t-test, ANOVA, correlation), and multivariate

statistics (e.g., multiple regression, ANCOVA, MANOVA,MANCOVA, GLM, survival, path analysis, SEM). Under-standing the hierarchy of statistical tests and the relatedassumptions is essential. Placement on the Arrow Frame-work is intended to represent these designs and statisti-cal analyses in their hierarchical context. More details re-lated to each analytic method can be found in standardresearch, statistics, and qualitative research methods text-books (Denzin & Lincoln, 2011; Grove et al., 2013; Polit& Beck, 2012; Tabachnick & Fidell, 1996).

Mixed methods design

Mixed methods design is represented at the center ofthe Arrow Framework, overlapping qualitative and quan-titative designs (see Figure 8), with curved symbols rep-resenting the flow of designs from each paradigm. Thedepiction oversimplifies mixed methods design, but pro-vides a basic visual display to indicate that mixed meth-ods design involves components from each paradigm.Creswell and Plano-Clark (2011) identifies six mixedmethod designs: three basic (i.e., Convergent, ExplanatorySequential, Exploratory Sequential ) and three advanced(i.e., Embedded, Transformative, and Multiphase). Mixedmethods design includes different combinations of pat-terns with different emphasis on research paradigms.For example, sequential mixed methods design empha-sizes a quantitative design followed by a qualitative com-ponent (QUANT -> qual), a primary qualitative designfollowed by quantitative component (QUAL -> quant),or a balanced, simultaneous design with equal empha-sis (QUANT<->QUAL; Creswell, 2012, 2013; Tashakkori& Teddlie, 2003a, 2003b). For discussion on mixedmethods design, the visual display can be expanded to

8


Step Two



RCTExperimental



Descriptive

Research Designs




Data AnalysisNarrative Numeric


Step One Frequencies (Nominal, Ordinal)Descriptives Central Tendency (Interval, Ratio)

SEMCausal Modeling/Path Analysis

Survival AnalysisLogistic Regression

Multiple RegressionGeneral Linear Model

ANCOVA, MANCOVAANOVA, MANOVAT-testCorrelation

Chi-Square

Meta-Synthesis Critique&AnalysisDiscourse AnalysisPhenomenological AnalysisGrounded Theory AnalysisEthnographic AnalysisNarrative AnalysisInterpretive Analysis

DescriptiveContent Analysis

Categories, Themes, & Patterns

Qualitative Analysis Multivariate & Inferential Statistics

Figure 7 Data analysis across research paradigms and research designs.

Major Research ParadigmsResearch Designs

MIXED METHODS QUANTITATIVEQUALITATIVE

Figure 8 Display of mixed methods with other major paradigms.

include several arrows depicting the varying forms ofmixed methods sequencing options. Selection of the typeand sequence of mixed methods designs is dependenton the research questions posed by the investigator, butmixed method design for research is used increasingly,and can maximize the strengths in each paradigm anddesign, while minimizing the weaknesses (Creswell &Plano-Clark, 2006; Morse & Niehaus, 2009; Sandelowski,2000a; Tashakkori & Teddlie, 2003a, 2003b).

Until students have a basic understanding of the quan-titative and qualitative research paradigms, and overallresearch designs, the introduction of mixed method de-sign research adds a layer of complexity that is poten-tially confusing for students. Thus, mixed method designis best discussed in detail following discussion of quantita-tive and qualitative designs, because use of mixed methoddesign integrates design based on contribution from qual-itative and quantitative designs and techniques.

9


Application of framework

The Arrow Framework has been used effectively inface-to-face classroom teaching, as well as in both syn-chronous and asynchronous online, distance-accessibleresearch methods courses. The framework is easily repli-cated in basic nonanimated or animated PowerPointslides and documents for teaching similar to the figuresin this article, adding the layers of complexity to illus-trate the interrelationships among concepts as needed forinstruction. For use in application-type exercises that areespecially critical for adult learners (Knowles, Holton, &Swanson, 2011; Merriam, 2007), the Arrow Frameworkcan be used following discussion regarding a particularfacet of research methods. For example, walking studentsstep-by-step through a critique of a published researcharticle is useful, with integrated use of the Arrow Frame-

work for each section of the article, and again as the cul-minating point of the critique. Queries can be posed tofacilitate student understanding, such as What research de-

sign (or sampling design, etc.) did the investigators use in thisresearch, or where exactly on the arrow would the design re-ported in this article fall? Or, the investigators did not iden-

tify the design used in their reported study—what are the designcharacteristics found in the report, and where would those char-acteristics lead you to place the design on the arrow? Follow-up with questions regarding the rationale for a student’sdecision to select placement at a particular point on theframework helps the student to articulate the specific cri-teria used in their decision making, moves the studentto application within the Arrow Framework, and iden-tifies the relevance to their own scholarly or practiceendeavors.

Student evaluation comments regarding use of the Ar-

row Framework have been very positive. For example, thearrow used as an instructional method was really easy tofollow (Fall Semester 2010); Most importantly, the arrowhelped me to read and understand published research(Fall Semester 2011); and Visual aids like the arrow reallyhelped to understand the material (Fall Semester 2011).Students have corresponded with faculty following grad-uation regarding the usefulness of the Arrow Frameworkas well, for example, “I thought of you today when I wasreading a research article: THE ARROW. I can’t thank youenough for helping to make sense of it all” (Pearce, 2012,personal communication ).

Examples

For those students who are experienced clinicians,making the linkage between clinical evidence and re-search, as well as evidence-based practice, is a criticalcomponent for their understanding that evidence is im-

portant and useful for their practices and their scholarlyendeavors. If teaching and learning are to be effectiveand students are to move to the next level of under-standing and application of what they have learned, it isthe responsibility of faculty teaching research methods toprovide appropriate and logical scaffolding examples thatlink the learner experience with the new material beingintroduced (Goldstein, 1999; Vygotsky, 1978; Vygotsky& Kozulin, 1986). But, that next level of understandingcan only occur if the students understand basic tenets ofdata or evidence. Students are cautioned to use clinicalanalogies carefully, because clinically relevant examplesand clinical data are not gathered necessarily for researchpurposes. The following examples are designed to helpstudents generalize their clinical knowledge to the largerdomain of research and evidence.

Examples such as the procedures for collectingand evaluating a laboratory result (e.g., CBC, lipids,hemoglobin A1C), or evaluative radiology tests (e.g.,chest x-ray, ultrasound), and safety in medication ad-ministration are easily adaptable examples relevant toall nurses regardless of educational program or specialty.These procedures involve a required level of overall rigorfor related activities, such as patient-preparation proce-dures, fasting parameters, and timing of blood draws, thatif not monitored will culminate in spurious results. Age ofthe patient can determine how a particular procedure iscompleted, and how the result is interpreted, given vary-ing context and patient characteristics (e.g., age, gender).Inter-rater and intrarater reliability can be exemplified withexplanations and probing questions, such as “if there isno specific, detailed protocol for completing a blood draw,then how can there be any assurance that the blood drawresult is valid or reliable?” When describing instrument-related reliability and validity, a corollary can be made tolaboratory results and how norms are developed, how lab-oratory equipment is calibrated, and how results are re-ported and interpreted.

An additional example that is universally understoodby all nurses, and quite useful in discussing reliabilityand validity of instruments, is the procedure of bloodpressure (BP) measurement. Discussion of the equipmentaddresses aneroid, mercury-based, and digital sphygmo-manometer measurement techniques, as well as arteriallines and more invasive procedures. For standard settings,it is helpful to include the notion of a gold standard that bywhich all other measurements or measurement tools arecompared. Mathematical models (e.g., regression mod-els) serve as a basis for programming digital techniques.However, those techniques are heavily reliant on years ofdata and clinical experience generated with arterial lines,mercury-based, and aneroid-based BP measurement. Va-lidity of measurement (does the [type] sphygmomanometer

10


Table 1 Helpful hints for using the Arrow Framework for educators and students

Educators Students

• Build the Arrow Framework in sequential order for PowerPoint

presentations, handouts, and discussion

• Use the Arrow Frameworkwhile you are reading about designs—think

about where the design fits best on the Arrow

• Emphasize the points on the Arrow Framework you are highlighting by

using a darker or colored font to mark specific point

• Emphasize vocabulary and terms in relationship to the language of

research, noting the position of the terms in application to designs on

the Arrow Framework

• Explain the horizontal and vertical placement of the designs on the

Arrow Framework in relationship to level of data and level of evidence.

Expand to add further designs as needed

• Highlight labels for the designs, so that students become familiar with

traditional labeling

• Discuss design (e.g., experimental), providing the characteristics or

assumptions that match the design

• Highlight information in context to the design lower or higher in

complexity to demonstrate the “stepping” of requirements to meet

design assumptions

• Use a published article for presentation and discussion of the specific

design, identifying the research question and/or hypothesis, and then

the specific design

• Discuss the specific characteristics of the design and ask students to

identify where on the arrow the design would be located

• Encourage students to apply the Arrow Framework in their readings of

research

• Provide examples that they can check and exercises that specifically

use published literature

• Focus on the design that is being presented on the Arrow Framework.

Note the level of data and level of evidence for the specific design in

relation to other designs

• Focus on understanding the vocabulary of research, noting the positionof terms in relation to the designs on the Arrow Framework. This will

help to learn the language in application to the designs

• Consider the horizontal and vertical axis of the Arrow Framework that

move from least complex to more complex designs, which parallel

level of data and level of evidence

• Learn the design labels—these are labels that are reflected in published

literature, and each includes particular activities, level of data, and

procedures overall

• Each design has specific characteristics that are hallmarks of the

specific design. These characteristics will come from your readings

and can be viewed against the Arrow Framework

• Think of each design in relationship to a design lower and higher to it in

complexity on the Arrow Framework, comparing the differences in

characteristics. You will find the characteristics build as you move

through the designs

• Think about the Arrow Frameworkwhile you are reading a published

research report

• Identify the research question and/or hypothesis, and identify the

location on the Arrow Frameworkwhere the design would fall. This will

help you to understand level of design complexity, level of data, and

level of evidence

• When reading a published research report, identify where on the

continuum the research should be on the Arrow Framework. You may

want to draw the Arrow in the white space of the paper and mark with

a star or other symbol. This will help you gather you about where in the

overall hierarchy the design is positioned, and will help you know what

to expect when you make critical judgments

actually measure what is purported to measure?), as well asreliability (e.g., can we depend on this measurement tool to al-ways, measure the same way and produce the same result, andwhat about the individual doing the BP measurement—is eachnurse doing it the same way?) are linked as clinical exam-ples, but used to exemplify the importance of reliable andvalid measurements in research . Examples from profes-sional experience are particularly useful in helping stu-dents reframe the notion of evidence, research, and re-lated parameters, and are especially helpful to make theseconcepts more relevant.

Medication safety in an acute care setting is an ex-ample used to explain the introduction of bias, orerror, with universal understanding and appeal for nurses.Using detail of all steps in medication administration,from the initiation of an order through pharmacy han-dling, delivery, and medication administration, includ-ing the people and departments involved is a directlyapplicable example of the many points at which er-

ror can occur (Bell, Cretin, Marken, & Landman, 2004;Miller & Bovbjerg, 2002). Queries, such as at what pointmight the possibility of error be introduced, and what is ran-

dom error and what is systematic error?, are more easilylinked with examples drawn from their professional workexperiences.

Conclusion

It is critical for students to understand the scientificunderpinnings of nursing. Science is based necessarilyon the use of evidence. Use of the Arrow Framework forteaching and learning research design has been demon-strated to be effective and practical as a model for teach-ing research methods to MSN, DNP, and PhD studentsby four faculty across five universities (see Table 1). TheArrow Framework provides a conceptual and practical ped-agogical approach to providing students a model for un-derstanding the varying aspects of research methods and

11


helps students understand the language of research, hierar-chy of evidence, and the application of the complex infor-mation more easily than rote memorization. Use of theArrow Framework improves the student’s ability to learnhow to critique and synthesize research evidence andtranslate the research concepts to their courses, scholarlyprojects, and practice. To that end, the use of the ArrowFramework provides a powerful scaffolding experience,and empowers students in learning difficult and complexresearch content, making sense of the evidence, and see-ing relevant application to practice and their scholarlyendeavors. Further, the Arrow Framework provides edu-cators a visual display that assists in the organization ofresearch content in a way that makes sense to studentsand enhances learning and application of research con-cepts. Moreover, the Arrow Framework provides a practical

solution for teaching complex research content that helpsstudents make the critical linkages between concepts andunderstanding of evidence in a straightforward manner.In this way, student learning about the scientific under-pinnings of nursing is enhanced and a better understand-ing of the complexity of research and the use of evidenceis achieved.

References

American Association of Colleges of Nursing (2001). Indicators of quality in

research-focused doctoral programs in nursing (position statement). Retrieved

from http://www.aacn.nche.edu/publications/positons/qualityindicators.

htm

American Association of Colleges of Nursing (AACN) (2006). Essentials of

master’s education in nursing. Retrieved from http://www.aacn.nche.edu

/education-resources/MastersEssentials11.pdf

American Association of Colleges of Nursing (AACN) (2011). Essentials of

doctoral education for advanced nursing practice. Retrieved from http://www.

aacn.nche.edu/publications/position/DNPEssentials.pdf

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching,

and assessing: A revision of Bloom’s taxonomy of educational objectives (complete

ed.). New York, NY: Longman.

Bell, D. S., Cretin, S., Marken, R. S., & Landman, A. B. (2004). A conceptual

framework for evaluating outpatient electronic prescribing systems based

on their functional capabilities. Journal of the American Medical Informatics

Association, 11(1), 60–70. doi:10.1197/jamia.M1374M1374 [pii]

Benner, P. E. (2010). Educating nurses: A call for radical transformation (1st ed.).

San Francisco, CA: Jossey-Bass.

Bloom, B. S. (1974). Taxonomy of educational objectives: The classification of

educational goals. New York, NY: McKay.

Chang, Y., Voils, C. I., Sandelowski, M., Hasselblad, V., & Crandell, J. L.

(2009). Transforming verbal counts in reports of qualitative descriptive

studies into numbers. Western Journal of Nursing Research, 31(7), 837–852.

doi:0193945909334434 [pii]10.1177/0193945909334434

Creswell, J. W. (2012). Six mixed methods designs for health and social science

research. Paper presented at the UAB Lister Hill Center for Health Policy:

Research Methods Workshop, Birmingham, AL.

Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed

methods approaches (4th ed.). Thousand Oaks, CA: Sage Publications.

Creswell, J. W., & Plano-Clark, V. L. (2006). Designing and conducting mixed

methods research. Thousand Oaks, CA: Sage Publications.

Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed

methods research (2nd ed.). Los Angeles, CA: Sage Publications.

Denzin, N. K., & Lincoln, Y. S. (2011). The Sage handbook of qualitative research

(4th ed.). Thousand Oaks, CA: Sage Publications.

DiCenso, A., Guyatt, G., & Ciliska, D. (2005). Evidence-based nursing: A guide to

clinical practice. Toronto, Ontario, Canada: Elsevier.

Dossey, B. M. (2010). Florence Nightingale: Mystic, visionary, healer

(commemorative ed.). Philadelphia, PA: F. A. Davis.

Fineout-Overholt, E., Melnyk, B. M., & Schultz, A. (2005). Transforming

health care from the inside out: Advancing evidence-based practice in the

21st century. Journal of Professional Nursing, 21(6), 335–344.

doi:S8755–7223(05)00145–6 [pii]10.1016/j.profnurs.2005.10.005

Forehand, M. (2005, 2010). Bloom’s taxonomy: Original and revised. In

M. Orey (Ed.), Emerging perspectives on learning, teaching, and technology.

Retrieved from http://projects.coe.uga.edu/epltt/

Funk, S. G., Tornquist, E. M., & Champagne, M. T. (1995). Barriers and

facilitators of research utilization: An integrative review. Nursing Clinics of

North America, 30(3), 395–407.

Goldstein, L. S. (1999). The relational zone: The role of caring relationships in

the co-construction of mind. American Educational Research Journal, 36(3),

647–673.

Grove, S. K., Burns, N., & Gray, J. (2013). The practice of nursing research:

Appraisal, synthesis, and generation of evidence (7th ed.). St. Louis, MO:

Elsevier/Saunders.

Hall, H. R., & Roussel, L. A. (2014). Evidence-based practice: An integrative

approach to research, administration, and practice. Burlington, MA: Jones &

Bartlett Learning.

Harris, R. L. (1996). Information graphics: A comprehensive illustrated

reference—Visual tools for analyzing, managing, and communicating. Atlanta,

GA: Management Graphics.

Institute of Medicine (U.S.). (2010). The future of nursing: Leading change,

advancing health. Retrieved from http://www.thefutureofnursing.org

Jones, S. C., Crookes, P. A., & Johnson, K. M. (2011). Teaching critical

appraisal skills for nursing research. Nurse Education in Practice, 11, 327–332.

doi:10.1016/j.nepr.2011.03.002

Kelly, K. P., Turner, A., Gabel Speroni, K., McLaughlin, M. K., & Guzzetta, C.

E. (2013). National survey of hospital nursing research, part 2: Facilitators

and hindrances. Journal of Nursing Administration, 43(1), 18–23.

doi:10.1097/NNA.0b013e3182786029

Knowles, M. S., Holton, E. F., & Swanson, R. A. (2011). The adult learner: The

definitive classic in adult education and human resource development (7th ed.).

Boston, MA: Elsevier.

McLaughlin, M. K., Gabel Speroni, K., Kelly, K. P., Guzzetta, C. E., & Desale, S.

(2013). National survey of hospital nursing research, part 1: Research

requirements and outcomes. Journal of Nursing Administration, 43(1), 10–17.

doi:10.1097/NNA.0b013e3182786045

Melnyk, B. M., & Fineout-Overholt, E. (2011). Evidence-based practice in nursing

& healthcare: A guide to best practice (2nd ed.). Philadelphia, PA: Wolters

Kluwer/Lippincott Williams & Wilkins.

Melnyk, B. M., Fineout-Overholt, E., Fischbeck Feinstein, N., Li, H., Small, L.,

Wilcox, L., & Kraus, R. (2004). Nurses’ perceived knowledge, beliefs, skills,

and needs regarding evidence-based practice: Implications for accelerating

the paradigm shift. Worldviews: Evidence Based Nursing, 1(3), 185–193.

doi:WVN04024 [pii]10.1111/j.1524–475X.2004.04024.x

Merriam, S. (2007). Learning in adulthood: A comprehensive guide (3rd ed.). San

Francisco, CA: Jossey-Bass.

Miller, R. H., & Bovbjerg, R. R. (2002). Efforts to improve patient safety in

large, capitated medical groups: Description and conceptual model. Journal

of Health Politics, Policy and Law, 27(3), 401–440.

Morse, J. M. (1999). Myth #93: Reliability and validity are not relevant to

qualitative inquiry. Qualitative Health Research 9, 717–718. doi:

10.1177/104973299129122171

Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002).

Verification strategies for establishing reliability and validity in qualitative

research. International Journal of Qualitative Methods, 1(2), 1–19.

12


Morse, J. M., & Field, P.-A. (1995). Qualitative research methods for health

professionals (2nd ed.). Thousand Oaks, CA: Sage Publications.

Morse, J. M., & Niehaus, L. (2009). Mixed method design: Principles and

procedures. Walnut Creek, CA: Left Coast Press.

Polit, D. F., & Beck, C. T. (2012). Essentials of nursing research: Generating and

appraising evidence for nursing practice (9th ed.). Philadelphia, PA: Wolters

Kluwer/Lippincott/Williams & Wilkins Health.

Sandelowski, M. (1986). The problem of rigor in qualitative research. ANS.

Advances in Nursing Science, 8(3), 27–37.

Sandelowski, M. (1993). Rigor or rigor mortis: The problem of rigor in

qualitative research revisited. ANS. Advances in Nursing Science, 16(2), 1–8.

Sandelowski, M. (2000a). Combining qualitative and quantitative sampling,

data collection, and analysis techniques in mixed-method studies. Research

in Nursing and Health, 23(3), 246–255.

Sandelowski, M. (2000b). Whatever happened to qualitative description?

Research in Nursing and Health, 23(4), 334–340.

Sandelowski, M. (2001). Real qualitative researchers do not count: The use of

numbers in qualitative research. Research in Nursing and Health, 24(3),

230–240.

Sandelowski, M. (2010). What’s in a name? Qualitative description revisited.

Research in Nursing and Health, 33(1), 77–84. doi:10.1002/nur.20362

Schmidt, N. A., & Brown, J. M. (2011). EBP for nurses: Appraisal and

application research (2nd ed.). Sudbury, MA: Jones & Bartlett Publishing.

Spoelstra, S. L., & Robbins, L. B. (2010). A qualitative study of role transition

from RN to APN. International Journal of Nursing Education Scholarship, 7(1),

1–15. doi:http://dx.doi.org/10.2202/1548–923X.2020

Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011). Individual

determinants of research utilization by nurses: A systematic review update.

Inplementation Science, 6, 1–14. doi:10.1186/1748–5908–6–1

Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.).

New York, NY: Harper Collins.

Tashakkori, A., & Teddlie, C. (2003a). Handbook of mixed methods in social and

behavioral research. Thousand Oaks, CA: Sage Publications.

Tashakkori, A., & Teddlie, C. (2003b). Issues and dilemmas in teaching

research methods courses in social and behavioural sciences: US

perspective. International Journal of Social Research Methodology: Theory &

Practice, 6(1), 61–77.

Tufte, E. R. (1990). Envisioning information. Cheshire, CT: Graphics Press.

Tufte, E. R. (2001). The visual display of quantitative information (2nd ed.).

Cheshire, CT: Graphics Press.

Vygotsky, L. S. (1978). Mind in Society: The development of higher psychological

processes. Cambridge, MA: Harvard University Press.

Vygotsky, L. S., & Kozulin, A. (1986). Thought and language. Cambridge, MA:

MIT Press.

Whittemore, R., Chase, S. K., & Mandle, C. L. (2001). Validity in qualitative

research. Qualitative Health Research, 11(4), 522–537.

13

Documents

Research methods for graduate students: a practical framework to guide teachers and learners