Mixed methods research: Toward new research designs in applied linguistics

Mixed Methods Research:Toward New Research Designs inApplied LinguisticsMOHAMMAD R. HASHEMIKharazmi University49, Mofateh StreetTehran 15614, IranEmail: [email protected]

ESMAT BABAIIKharazmi University49, Mofateh StreetTehran 15614, IranEmail: [email protected]

Despite criticisms questioning its raison d’etre (e.g., Giddings & Grant, 2007), mixed methods researchhas been welcomed in social research as a methodology in its own right (Greene, 2008). Recently, it hasalso been acknowledged and advocated in applied linguistics (Dornyei, 2007; Hashemi, 2012). In anattempt to investigate the status of this relatively new trend in applied linguistics, the current studyexamines the nature of the integration of qualitative and quantitative methods in terms of researchdesigns, sampling designs, and quality of interpretations. Content analysis of 205 research articlespublished in seven comprehensive international peer reviewed applied linguistics journals between 1995and 2008 reveals that concurrent designs are more prevalent than sequential designs and that studiesmake limited use ofmixed designs that are detailed in themixedmethods literature.Moreover, althougha considerable number of articles used both qualitative and quantitative methods, only a small numberachieved high degrees of integration at various stages of the study as a quality standard for mixedresearch. The study concludes with several implications for makingmore effective use of mixedmethodsresearch in applied linguistics and calls for a more systematic treatment of this trend as a versatileresearch methodology.

Keywords: inference quality; mixed methods research; qualitative; quantitative; research design

OVER THE PAST DECADES, INQUIRY INapplied linguistics has undergone considerablemovement in response to the academic require-ments of the field (see Cumming, 1994;Davis, 1995; Lazaraton, 1995, 2000). Among theforces driving this shift were advances in thenatural sciences and developments and chal-lenges in social and educational research(Dornyei, 2007). One particularly controversialissue has been the dichotomy between qualitativeand quantitative research methods, which hasbeen singled out for criticism by several scholars(Newman & Benz, 1998; Ridenour & Newman,2008; Sale, Lohfeld, & Brazil, 2002; Sechrest &Sidana, 1995; Tarrow, 2004). In time, these

controversies led to the emergence of a newapproach in social and educational researchcalled mixed methods research (Greene, 2007,2008; Johnson & Onwuegbuzie, 2004; Tashakkori& Teddlie, 1998). While this approach hasattained considerable prominence in social andeducational research, it does not seem to havegained the kind of attention in applied linguisticsresearch that we believe it deserves.

To be sure, the nature, occurrence, andrespective merits of qualitative and quantitativemethods in applied linguistics have recentlyreceived considerable attention (see e.g., Bensonet al., 2009; Davis, 1995; Lazaraton, 1995, 2000,2002, 2003, 2005; Norton, 1995; Plonsky & Gass,2011). However, remarkably little discussion hasaddressed how the twomethods might usefully beintegrated in a way that would enhance theresearch tools and, by extension, the findings thatmight be obtained in applied linguistics research.As a first step to encourage such a move, the

The Modern Language Journal, 97, 4, (2013)DOI: 10.1111/j.1540-4781.2013.12049.x0026-7902/13/828–852 $1.50/0© 2013 The Modern Language Journal

present study was designed to accomplish twogoals: first, to investigate the current status ofmixed methods research in applied linguisticsand, second, to explore the nature of mixedresearch designs, sampling designs, and thequality of inferences that can be obtained withmixed methods research (Tashakkori & Teddlie,2008).

REVIEW OF THE LITERATURE

Mixed Methods Research: Definition and Origins

Bergman (2008) provides a general definitionof mixed methods research as “the combinationof at least one qualitative and at least onequantitative component in a single researchproject or program” (p. 1). However, it wouldbe simplistic to assume that merely utilizing bothqualitative and quantitative components would inand of itself result in mixed methods research(see Bryman, 2008). In a somewhat more techni-cal sense, then, mixed methods research can bedefined as “the collection, analysis, and integra-tion of quantitative and qualitative data in a singleormultiphase study” (Hanson et al., 2005, p. 224).The importance of the presence of mixing at allstages in a research project is reflected in a yetmore extended definition of mixed methodsresearch provided by Tashakkori and Creswell(2007a): “research in which the investigatorcollects and analyzes data, integrates the findings,and draws inferences using both qualitative andquantitative approaches in a single study orprogram of inquiry” (p. 4). That is to say, high-quality mixed methods research requires mixingthroughout an entire study: from formingresearch questions, to sampling, to data collec-tion, to analysis, and, finally, to interpretation(Yin, 2006).

Mixed methods research has evolved frommono method research (Tashakkori & Teddlie,1998). Although the first instances of combiningqualitative and quantitative methods can betraced back to the 1800s (Hesse–Biber, 2010;for amore detailed historical account, see Teddlie& Tashakkori, 2009), the recent history of mixingmethods has been influenced by the emergenceof the concept of triangulation introduced byCampbell (1953; Campbell & Fiske, 1959). Theterm “triangulation” comes from navigation andmeans “taking bearings on two landmarks inorder to locate one’s position” (Hammersley,2008, p. 23). This method is also used in landsurveying to “determine the position of a thirdpoint C if we know the positions of points A and B,

and we also know the interior angles (X and Y) atany two points of the triangle thus formed by A, B,and C” (Gorard, 2004, p. 44). Originally, it mayhave appeared that the notion of “triangulationreferred only to the use of multiple forms ofqualitative research methods, not the combina-tion of quantitative and qualitative methods”(Denzin, 2012, p. 3). However, triangulationwas among the major sources of influence fromwhich the idea of mixing qualitative and quanti-tative methods emerged and evolved in socialresearch. More recently, qualitative methodolo-gists in favor of richly developed, multidimen-sional, multigeneric descriptions of the worldhave deconstructed and further reread theconcept of triangulation in light of another newconcept, that of crystallization (see Ellingson, 2009,2011).

Theoretical Issues in Mixed Methods Research

Much of the theoretical debate about mixedmethods research focuses on the possibility ofmixing at the paradigmatic level. The mainproblem raised in such discussions addressesthe incompatibility thesis (Johnson&Onwuegbuzie,2004) and commensurability validity (Onwuegbuzie& Johnson, 2006); their focus is to explorewhether integrating qualitative and quantitativemethods is possible given the dissimilar ontologi-cal and epistemological stances that influenceeach method. To address this problem, scholarsin the field have presented different positions,such as the pragmatist position (Datta, 1997;Feilzer, 2010; Johnson & Onwuegbuzie, 2004;Morgan, 2007), the dialectical position (Johnson,2008), the balanced view (Greene & Caracelli,1997), the transformative paradigm (Mertens, 2007,2010), and the critical interpretive view (Denzin,2012). Taken together, these positions hold, froma practice-oriented perspective, that the problemof incommensurable philosophical assumptionsbehind the two approaches need not be a majorhindrance; in any case, the practical value ofmixing is quite evident. In fact, experts inresearch methodology increasingly believe thatqualitative and quantitative methods can coexistin a dialectical relationship and that findings fromthe two strands, being convergent or divergent,enhance the understanding of the phenomenonunder study. In short, research would beinformed by both approaches which, together,contribute to the nature of inquiry in social andeducational research. In reverse, absent one orthe other, many questions would remain onlypartially answered or even wholly unanswered.

Mohammad R. Hashemi and Esmat Babaii 829

With regard to this issue, then, progress inscientific inquiry would not occur in the form ofthe famous Kuhnian displacement of paradigms;rather, this might be one of those rare circum-stances “under which two paradigms can coexistpeacefully” (Kuhn, 1962, p. ix).

In addition to the debates on paradigmaticproblems and orientations in mixed methodsresearch, a number of contributions have consid-ered other theoretical and conceptual aspects ofmixed research, with the goal of presentingcredible mixed methods models and frameworks.These discussions range over a number of issues,such as a critical analysis of the discourse of mixedmethods research (Freshwater, 2007), a particularfocus on the strengths of both qualitative andquantitative methods in a synergistic approach (Hall& Howard, 2008), the significance of the socialaspect of mixed research and the concept ofcommunities of practice (Denscombe, 2008), theimportance of the human element (Tashakkori &Teddlie, 2010), the application of the theory ofreflexive sociology to mixed methods inquiry(Fries, 2009), and the existence of variousparadigms within a metaparadigmatic realizationof mixed research (Harrits, 2011).

Methodological Issues in Mixed Methods Research

Bazeley (2004) addressed several methodologi-cal concerns that have been raised in conjunctionwith mixing qualitative and quantitative researchand warned against the danger of the “corruptionof methods” (p. 144). According to her, mixedmethods research is more than just using a varietyof methods or using one method in support ofanother; rather, mixed methods research com-bines “nomothetic and idiographic approaches inan attempt to serve the dual purposes ofgeneralisation and in-depth understanding—togain an overview of social regularities from alarger sample while understanding the otherthrough detailed study of a smaller sample”(p. 145). Similarly, Yin (2006) raised the questionof whethermixingmethodsmeans usingmethodsin parallel in a single study or integratingmethodsto enhance the integrity of the study. By specifyingthe pitfalls in designing and conducting mixedresearch, he asserts that researchers should“tighten the use of mixed methods so that theydo in fact occur as part of a single study” (p. 42,original emphasis). They would accomplish this—and, by implication, high-quality mixed methodsresearch—by specifying the nature of the integra-tion that would take place with regard to researchquestions, units of analysis, samples for study,

instrumentation and data collection methods,and analytic strategies.

The aforementioned levels in mixed methoddesigns as well as mixed method validity issueshave been explored in several studies.With regardto mixed methods research questions, scholarssuggest that they be formulated on the basis of theresearch purpose; they might range from over-arching mixed questions to parallel and sequen-tially formed questions (Creswell & PlanoClark, 2007; Onwuegbuzie & Leech, 2006; Ta-shakkori & Creswell, 2007b;Woolley, 2009). As formixed methods research designs, as explored insocial research, the advocated research typologiestake account of two major categories: concurrentand sequential mixed designs (Creswell, PlanoClark, & Garrett, 2008; Hanson et al., 2005;for a critical appraisal of mixed designs andan alternative to typologies, see Guest, 2013).Particularly notable novel mixed designsinclude extended-term mixed method (ETMM) evalua-tion designs (Chatterji, 2004)-ETMM designscombine descriptive research methods andexperimental designs in different phases of aproject to “follow life-spans of individualprograms/policy initiatives” (Chatterji, 2004,p. 15), the heuristic model ofmixed research designs(according to Nastasi et al., 2007, p. 165, suchdesigns utilize “a recurring sequence of qualitativeand quantitative data collection culminating in arecursive qualitative–quantitative process”), andthe recursive mixed methods approach (Christ, 2007;this design was used by Christ who integrated anexploratory quantitative phase, an exploratorycross-case analysis, and an exploratory longitudi-nal phase in a sequential design). Research intothe nature of mixed methods sampling designs,the third category mentioned by Yin, has shownthe prevalence of concurrent and sequentialdesigns (Collins, Onwuegbuzie, & Jiao, 2006,2007; Teddlie & Yu, 2007). Not surprisingly, arange of mixed methods data collection andanalysis strategies have been explored and evalu-ated. In their pioneering work, Caracelli andGreene (1993) proposed integrative data analysisstrategies including “data transformation,” “typol-ogy development,” “extreme case analysis,” and“data consolidation/merging” (pp. 197–199).Another notable example in program evaluationis the flexible judgment technique (McConney, Rudd,& Ayres, 2002), which has two main functions: (a)it “rates (and thereby weights) each piece ofqualitative or quantitative evidence using ‘criteriaof worth’ (CoW) aligned with the aims of theprogram under evaluation,” and (b) it also“provides a method whereby they can be

830 The Modern Language Journal 97 (2013)

aggregated to assist evaluators in reaching adefensible conclusion” (McConney et al., 2002,p. 124). A third strategy is termed nested analysis(Lieberman, 2005); serving as a unified approachto comparative analysis, this strategy involves thecombination of “the statistical analysis of a largesample of cases with the in-depth investigation ofone or more of the cases contained within thelarge sample” (Lieberman, 2005, pp. 435–436).Other mixedmethods data collection and analysisstrategies include mixed mode survey research (whichis a multiphase data collection strategy; Leeuw &Hox, 2008), crossover graphical displays (that, asexplained by Onwuegbuzie & Dickinson, 2008,p. 210, involves the “creation of unique visualelements to summarize and highlight importantdata characteristics and research implications”),and exploratory soft ontology or ESO (which, accord-ing to Niglas, Kaipainen, & Kippar, 2008, p. 172,draws on the “conceptualization of (meta)data” tointegrate a “structured way of” exploration with a“non-deterministic flexible approach to data”),and finally strategies such as data synthesis,intensive case analysis, comparative coding, pat-tern analysis, data conversion, blending of varia-bles, and iterative analysis for integrating dataanalysis procedures (Bazeley, 2009).

Finally, mixed methods validity issues havebeen considered by several researchers throughpresenting and evaluating validity frameworks.Onwuegbuzie and Johnson (2006), for example,elaborated on the importance of validity andcredibility of both quantitative and qualitativestrands in a mixed design. Introducing the termlegitimation, they argued that mixed methodsresearch combines the “complementary strengthsand “non-overlapping weaknesses” of both quali-tative and quantitative approaches; for this reasonassessing the “validity of findings” is complex(p. 48). Dellinger and Leech (2007) examineddifferent views on validity and introduced aunified validity framework for mixed methodsresearch. Their framework is a construct ofvarious interacting elements, including whatthey refer to as the “foundational element,”“inferential consistency,” the “utilization/histori-cal element,” and the “consequential element”(pp. 323–325). Tashakkori and Teddlie (2008)provide an integrative framework for assessinginference quality in mixed methods research. Theyuse the term “inference quality” to refer to the“quality of conclusions that are made on the basisof the findings in a study” (p. 101). They arguethat the uniqueness of mixed methods researchlies in the fact that it gathers “information that canresult in ‘meta-inferences’ about the phenome-

non under study that neither the quantitative norqualitative perspectives could do alone” (p. 101).

As for meta-inferences, they are “an overallconclusion, explanation, or understanding devel-oped through an integration of the inferencesobtained from the qualitative and quantitativestrands of a mixed methods study” (p. 101).Further elaborating on this point, Tashakkori andTeddlie (2003) had previously stated that aninference is “a researcher’s construction of therelationships among people, events, and varia-bles, as well as his or her construction of therespondents’ perceptions, behaviors, and feelingsand how these relate to each other in a coherentand systematic manner” (p. 692). In sum, validityclaims in mixed research can be developed inlight of inference quality that can be realized andevaluated based on an integrative framework (seeTashakkori & Teddlie, 2008, p. 102).

Although there is little systematic research intothe nature of mixed designs in applied linguisticsresearch, a goodnumber of studies have used bothqualitative and quantitative components. Amongthese, a small number seem to have utilizedmixedmethods research with high integration quality(for examples, see Barkaoui, 2010; Kim, 2009;Kissau & Wierzalis, 2008; Lamb, 2007; Lee &Greene, 2007; Tan, 2011; Zhang, 2011).

Considering the above methodological issuesin mixed methods research and given ourresearch purpose, we attempted to address mixedmethods research designs, sampling designs, andinference quality in mixed research in appliedlinguistics. To do so, the present study poses thefollowing research questions:

THE STUDY

Data Gathering

The data sources for this study constitute apurposive sample (see Benson et al., 2009;Bryman, 2008; Teddlie & Yu, 2007) of articlespublished in seven comprehensive internationalpeer reviewed applied linguistics journals. Thedata consisted of a corpus of articles publishedover a period of 14 years, between 1995 and 2008

RQ1. What kinds of research designs are usedwhen method mixing is utilized in appliedlinguistics research?

RQ2. What kinds of sampling designs are used inmixed methods research in the field ofapplied linguistics?

RQ3. How are general conclusions ormeta-inferences developed in this type ofresearch?


(with the exception of the journal of LanguageTeaching Research whose publication started in1997).

More specifically, the sampling scheme for thisstudy was multilayered (see Alise & Teddlie, 2010)regarding the selection of the journals and thechoice of specific articles. Journal selection wasbased on five sources: the list of professionaljournals published byTheModern Language Journal(Weber & Campbell, 2004), Egbert’s (2007)evaluation of applied linguistics journals, Jung’s(2004) examination of the frequency of appear-ance of ELT journals selected for presentation inLanguage Teaching between 1996 and 2002,Lazaraton’s sample (2000)—a study on the useof qualitative and quantitative methods in 332articles over a 7-year period, and expert judgment(i.e., two PhD holders with research experience ofmore than 10 years and currently active in thefield of applied linguistics). Using the first foursources yielded a comprehensive list of 64journals. To select a sample representing approx-imately 10% of all journals, we used expertjudgment. We consulted with two researcherscurrently active in the field of applied linguisticsto produce the final list. Among reasons forexcluding a journal were: (a) an indirect relation-ship to the field of applied linguistics (e.g., BritishJournal of Educational Psychology, Computers andComposition, International Journal of InterculturalRelations, Journal of Educational Measurement, etc.);(b) publishing in a language other than English(e.g., Etudes de Linguistique Appliquee); (c) insuffi-cient information in the methodology sectionof articles published in the journal (e.g., ELTJournal—see Benson et al., 2009); (d) lesserprominence on the international scene (e.g.,Korean Journal of Applied Linguistics); and (e)strongly delimited research focus (e.g., Journal ofSecond Language Writing, Assessing Writing, Lan-guage and Cognitive Processes). The seven journalschosen on the basis of this procedure were AppliedLinguistics (AL), English for Specific Purposes (ESP),Language Learning (LL), Language Teaching Re-search (LTR), Language Testing (LT), The ModernLanguage Journal (MLJ), and TESOL Quarterly(TQ).

The process of gathering the data sourcesproceeded in the following manner: First, theelectronic versions (Adobe Reader version 7) ofall of the articles published in these seven journalsfrom 1995 to 2008 were collected.We scanned theabstract and methodology sections of the articles,focusing on whether the design in each studyinvolved mixing at the stages of sampling, datacollection, and/or data analysis. We also used the

search function of Adobe Reader, searching forkey words or phrases such as mixed methods, multi-method, qualitative, quantitative, triangulation, inte-grating methods, and combining methods. In thisscreening phase, a total of 273 articles wereidentified.

Data Analysis

At the initial stage of the analysis, all 273articles were subjected to a rigorous qualitativecontent analysis. We sought to provide a richdescription1 (Erickson, 1986) of the content,particularly those parts that involved the descrip-tion of howmixing or combining took place in theprocess of data collection and analysis. Thisrigorous examination of the content was carriedout through an iterative process (for some articleswe read, examined, and reexamined the meth-odology section more than three times, highlight-ing the key points). We paid particular attentionto the design of the study, type of sampling, andprocesses of data collection and analysis. Also,arguments of validity and concluding interpreta-tions were considered and we noted whether theresearcher(s) made any argument(s) that couldbe indicative of using meta-inferences to ensureinference quality (Tashakkori & Teddlie, 2008)based on the results from integrating methods.

Similar to Alise and Teddlie (2010), analysisprocedures used in the present study were basedon both manifest content (the concrete terms usedfor coding) and latent content (the coder’sinterpretation of the content). Thus, to stream-line our work with the data (see Patton, 2002), inaddition to making descriptive and detailed notesof the contents of the articles, we made use ofcode sheets whose categories were developedbased on the following frameworks andtypologies:

Analysis of Research Designs was handled on thebasis of the methodological framework presentedby Creswell et al. (2008) who classified researchdesigns as either concurrent (i.e., designs that usequalitative and quantitative methods simulta-neously) or sequential (i.e., designs that makeuse of the methods separately, in different phasesof the research); they further presented a numberof subdesigns (see especially pp. 67–70): (a)Concurrent Triangulation Design (collection andanalysis of QUAN and QUAL data are conductedin parallel and integrated interpretations aredrawn based on QUAN and QUAL results); (b)Concurrent Embedded Design (QUAL data arecollected within the QUAN experimental design,between pretests and posttests, and integrated


interpretations are made based on QUAN andQUALdata); (c) Sequential Explanatory Design (firstQUAN data are collected and analyzed; then, tofurther explain the findings from the first phase,QUAL data are collected and analyzed; finally,integrated interpretations are drawn based onQUAN and QUAL data); (d) Sequential ExploratoryDesign (first QUAL data are collected andanalyzed; then, to further explore the phenome-non, QUAN data are collected and analyzed;finally, integrated interpretations are drawnbased on QUAL and QUAN data); and (e)Sequential Embedded Design (it “typically involvescollecting qualitative data before an interventionbegins or after it is complete” [p. 69]; integratedinterpretations are then made on the basis ofQUAL and QUAN data).

Sampling Designs were analyzed based on Collinset al.’s (2007) two dimensional model of mixedmethods sampling. The model presents “atypology in which mixed methods samplingdesigns can be categorized according to (a) thetime orientation of the components and (b) therelationship of the qualitative and quantitativesamples” (p. 276). Taking into account timeorientation, they divide sampling designs intoconcurrent and sequential categories. With re-gard to the relationship between qualitative andquantitative samples, they present four groups ofsampling designs for each of the above majorcategories—(a) identical: qualitative and quantita-tive samples include the same members; (b)parallel: different qualitative and quantitativesamples are drawn “from the same underlyingpopulation” (p. 277); (c) nested: “the samplemembers selected for one component of theinquiry represent a subset of those participantschosen for the other phase of the study” (p. 277);and (d) multilevel: two or more sample sets fromdifferent populations are used at “different levelsof the investigation” (p. 277).

Inference Quality was established by examiningthe discussion and conclusion sections of thearticles, using Tashakkori and Teddlie’s (2008)“integrative model of inference quality in mixedmethods research” (p. 112). This model involvestwo broad categories: design quality, includingwhat they refer to as design suitability, designadequacy, within design consistency, analytic adequacy;and integrative rigor, including the categories ofinterpretive consistency, theoretical consistency, interpre-tive agreement, interpretive distinctiveness, and inte-grative efficacy (pp. 112–116). It is important tostate that, although the authors attempted topresent conceptual definitions for these ele-ments, they did not succeed at presenting

operational definitions that include objectivelymeasurable indicators for them. In the presentstudy, the analysis was, thus, based on only one ofthe above components (integrative efficacy) toexamine “the degree to which inferences made ineach strand of mixed methods study are effective-ly integrated into a theoretically consistent meta-inference” (Tashakkori & Teddlie, 2008, p. 115);that is, whether the findings and inferences fromboth qualitative and quantitative strands areintegrated and presented as meta-inferences.

In light of this situation it is, then, notsurprising that it was not always readily apparentwhether the studies we investigated effectivelyintegrated qualitative and quantitative methods.Therefore, we initially based our analysis onBergman’s (2008) simple definition of mixingin order to be able to categorize articles withgreater ease. What made the scope narrower andthe task of content analysis more complicated wasthe use of the extended definition presentedearlier on (cf. Tashakkori &Creswell, 2007a). Thisdefinition asserts that mixing should happen atdifferent stages of a particular study and thatmeta-inferences must be developed based on dataintegration. Applying this more demandingstandard revealed that, of the 273 articles, 68had not actually integrated qualitative andquantitative methods. Rather, one approachinfluenced the methodology and componentsof the other approach (i.e., numbers, statistics,words, verbal description) were only used as a partof the main method. Thus the final sampleconsisted of 205 articles.

Cohen’s Kappa for Inter-Coder Reliability

To ensure reliability of the coding, an indepen-dent coder coded a sample of the researcharticles. The coder had a background in qualita-tive research and, before starting the process ofcoding, took part in a 2-hour briefing session tobecome familiar with mixed methods designs.About 10% of the sample (i.e., 20 articles) wascoded by this researcher. For this study, Kappacoefficients were 0.82 for the coding of researchdesigns and 0.71 for the coding of the samplingdesigns. These indices were checked againstAltman’s (1991) criteria and proved acceptable.

FINDINGS

Research Question 1

Table 1 presents the frequencies with whicheach design was used in the articles from the


sample of journals. The results of the chi-squaregoodness-of-fit test for the total frequenciesrevealed that there were significant differencesin the designs employed in the journal articles:x2¼ 379.4, df¼ 5, p< .001. Table 2, a summarytable, shows the correspondence between eachdesign type and the studies that used the design.

The overall percentages for the use of designsin the studies are quite different. As shown inTable 1, concurrent designs were used more

frequently (71.71%) than sequential designs(24.88%) and a small number of the studies(3.41%) used designs that did not exist in Creswellet al.’s (2008) classification. Among studies withconcurrent designs, a few used the concurrentembedded design. A good number of the studiesused triangulation (66.34%), the most frequentlyused design. Embedded designs in both concur-rent and sequential categories, on the other hand,were used with the lowest frequencies: 5.37% and

TABLE 1Frequencies of Mixed Methods Research Designs Used in the Articles

Journal

Concurrent Designs Sequential Designs

TotalTriangulation Embedded Explanatory Exploratory Embedded Other Designs

F P % F P % F P % F P % F P % F P % F

AL 23 79.31 0 0.00 3 10.34 2 6.90 0 0.00 1 3.45 29ESP 19 61.29 0 0.00 10 32.26 1 3.23 0 p0.00 1 3.23 31LL 16 59.26 5 18.52 3 11.11 2 7.41 0 0.00 1 3.70 27LT 24 72.73 0 0.00 2 6.06 4 12.12 2 6.06 1 3.03 33MLJ 28 60.87 6 13.04 7 15.22 3 6.52 1 2.17 1 2.17 46TQ 19 63.33 0 0.00 8 26.67 1 3.33 0 0.00 2 6.67 30LTR 7 77.78 0 0.00 0 0.00 1 11.11 1 11.11 0 0.00 9Total 136 66.34 11 5.37 33 16.10 14 6.83 4 1.95 7 3.41 205

Note. Abbreviations: AL:Applied Linguistics; ESP: English for Specific Purposes; LL:Language Learning; LT:LanguageTesting; MLJ: The Modern Language Journal; TQ: TESOL Quarterly; LTR: Language Teaching Research. F and P standfor frequencies and percentages of mixed methods research designs in the sample.Table 1 is an adapted version of a table that was originally published in a brief progress report of this study inApplied Linguistics, 33(2), (2012).

TABLE 2Mixed Methods Research Designs in the Data Sources

Mixed MethodsResearch Designs Data Sources

Concu

rren

t

Triangulation 1, 3, 4, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 26, 27, 28, 29, 30, 31, 33,35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 52, 55, 58, 59, 60, 62, 63,64, 65, 66, 69, 70, 71, 72, 76, 77, 80, 81, 82, 83, 84, 86, 89, 90, 92, 93, 94, 95, 96, 98,99, 100, 103, 104, 109, 110, 112, 113, 115, 118, 120, 122, 124, 125, 126, 127, 129,130, 132, 133, 135, 136, 137, 138, 139, 140, 141, 142, 146, 147, 148, 152, 153, 155,156, 160, 162, 163, 164, 166, 167, 168, 170, 171, 173, 177, 178, 179, 183, 184, 185,187, 189, 190, 191, 192, 194, 195, 196, 198, 199, 200, 201, 202, 205

ConcurrentEmbedded

5, 34, 105, 106, 119, 128, 149, 172, 175, 176, 180

Sequen

tial

Explanatory 2, 11, 12, 16, 23, 25, 32, 48, 53, 54, 57, 68, 73, 74, 75, 78, 79, 85, 87, 91, 97, 101,102, 114, 150, 158, 159, 161, 169, 174, 182, 188, 204

Exploratory 56, 67, 88, 107, 111, 116, 121, 123, 131, 134, 145, 151, 165, 181SequentialEmbedded

61, 108, 144, 193

Other OtherDesigns

117, 143, 154, 157, 186, 197, 203


1.95%, respectively. Similarly, the sequentialexploratory design was used in a limited numberof articles (6.83%). Finally, the design that wasmost common in the sequential category was theexplanatory design (16.10%).

In addition to the quantitative analysis of theresults, the data were explored in detail to presenta rich description of the emerging patterns. Thefirst notable issue is the way researchers addressedtheir designs and whether they used mixedmethods terminology to elaborate on the proce-dures.2 Although the use of mixed designs wasdirectly acknowledged in only a few studies andmixed methods research designs were rarelyreferred to, the term triangulation was used in alarge number of studies. Examples of this (e.g.,including key terms such as triangulation, multi-ple data sources, multiple measures, qualitativeand quantitative data) are presented in theextracts below. (In the following examples andextracts the numbers in the square brackets referto the data source provided in the data sourcebibliography in Appendix A).

[57] ( . . .) a triangulation of data collection proce-dures ( . . .) (Frank, 2000, p. 33)

[101] The study investigates, both quantitatively andqualitatively, personal pronouns in a corpus ofscientific journal articles. (Kuo, 1999, p. 123)

[122] We used different measures, data sources, andmethods. The combination of measures, data sour-ces, and methods not only allowed triangulation ofthe finding that ( . . .)

(McGroarty & Zhue, 1997, p. 2)

[160] In order to reveal teachers’ beliefs, knowledge,and practices about CLT, we employed triangulationthat included qualitative and quantitative datasources (or multiple data sources) of LOTE teachers’perspectives. (Sato & Kleinsasser, 1999, p. 499)

In several studies, the researchers used mixingonly at the analysis stage without acknowledgingthat they had integrated qualitative and quantita-tive methods. In such cases, a third type oftriangulation emerged in the process of dataanalysis. This type of triangulation can be referredto as triangulation by methods of analysis or, simply,triangulation by analyses (see Denzin, 1978). Sever-al researchers asserted that they used bothqualitative and quantitative methods of analysisso that one would complement the other:

[142] Two types of analysis were performed: ( . . .) aquantitative analysis ( . . .) a qualitative analysis ( . . .)

(Precht, 1998, p. 241)

[41] Data analysis for the survey involved bothquantitative and qualitative procedures ( . . .)

(Dong, 1998, p. 374)

[28] ( . . .) we have carried out a quantitative andqualitative analysis of actual corpus-based data ( . . .)

(Cruz et al., 1998, p. 52)

In a more detailed analysis of the data, itbecame evident that among the 136 articles in thedata sources that used triangulation in a mixeddesign, 63 used triangulation by analyses; that is,triangulation only at the level of data analysis.

Although scant mention of mixed methodsterminology was made in the studies, in a numberof articles (e.g., data sources [14], [102], [122],[157], [165], [199]) the researcher(s) used termssuch as multiple methods, multiple data analysis,mixed method, and mixed methods. This showsthat there is a growing awareness amongresearchers of the value of mixing qualitativeand quantitative methods.

Another important finding was the emergenceofmixed research designs that were not present inCreswell et al.’s (2008) typology. Of the 205studies, 7 (3.41%) used designs that were notsimilar to any of the designs in the concurrentand sequential categories. These were coded as“Other” on the code sheet.

The first new design that emerged in the codingprocess was used in Mackey (2006; data source[117]), where noticing and learning were ana-lyzed for each participant through a pretest–posttest design, using learning journals, oralstimulated recall protocols, written responses toa focused question, and written responses onquestionnaires. The experimental design used inthe study was particularly unique in that theexperiment included an embedded qualitativecomponent that was followed by another qualita-tive component. That is, learning journals weredeveloped between the pretest and posttestsessions in the treatment sessions. After theposttest, stimulated recall was used with theexperimental group. A qualitative component(learning journals) was used before the begin-ning of the experiment for familiarizing thelearners with journal writing, but this cannot beconsidered to be an actual phase of datacollection. Finally, responses to a focused ques-tion and also questionnaires were collected. Thedesign in the study can be depicted as follows:

Quan ðQual embeddedÞ ! Qual

The secondnewdesign appeared in data source[143], in the study by Pritchard & Nasr (2004).


This was a needs analysis of Egyptian engineeringstudents to design a reading improvement pro-gram. In the first phase of the study, a qualitativecomponent was used to assess the needs of thestudents. Based on the results from this phase, theresearchers used the materials in an experimentto further test their reliability and validity. Thefinal phase of the study after the experimentalmodule involved collecting both quantitative andqualitative feedback from teachers and studentsto evaluate the materials. The design of the studycan be presented as follows:

Qual ! Quan ! ðQuanþ QualÞSasaki (2004; data source [157]) presented the

third new design. This developmental study used“multiple data sources” (p. 525), including bothquantitative and qualitative components, andcollected data in four stages. In this sequentialprocess, both qualitative and quantitative datawere collected in parallel. This creates a newdesign that does not exist in Creswell et al.’s(2008) sequential category:

Quan ! Quan ! Quan ! Quan

Qual ! Qual ! Qual ! Qual

The fourth new design that emerged in theprocess of data analysis was used in data source[154], the study by Saif (2006), in which theresearcher used a “multiphase empirical study”(p. 1) to investigate the washback effects of a testof spoken language proficiency. The study wascarried out in three phases: (a) needs analysis(interviews, questionnaires, document analysis,syllabi analysis), (b) test development (identifyingthe TLU domain, developing the test), and (c)test consequences (using the test, collectingqualitative and quantitative data). This design issimilar to the third design in that both qualitativeand quantitative components were utilized bythe researcher in the three phases above. Soa triangulated combination of sequential andconcurrent designs was used:

ðQualþQuanÞ!ðQualþQuanÞ!ðQualþQuanÞ

The study byWilhelm (1997; data source [197])presented a fifth emerging design. It investigateda collaborative model of teacher training in twostages. In the first stage, the course was pilotedand qualitative as well as quantitative data werecollected. In the second stage, the revised course

was implemented and feedback was collectedboth qualitatively and quantitatively:

ðQualþ QuanÞ ! ðQuanþ QualÞSimilar to the previous design, the sixth new

design appeared in Vavrus (2002; data source[186]), which presents a combination of concur-rent and sequential designs in two stages:

ðQuanþ QualÞ ! ðQuanþ QualÞThe last new design came from Yu’s (2008)

study (data source [203]), which investigatedboth the process and product of EFL classroompractice and its effect on students’ developmentof sociolinguistic competence. The first stage ofthe study involved both qualitative and quantita-tive components (qualitative data were alsocollected in parallel to the experimental process).Then a posthoc qualitative analysis was con-ducted. This was used for interpreting thequantitative data. It seemed that the study hadcombined sequential and concurrent designs,representing a triangulated explanatory designwhich would look like the following:

ðQuanþ QualÞ ! Qual

Research Question 2

Results of the chi-square goodness-of-fit test forsampling designs indicated that there weresignificant differences in the sampling designsused in the articles: x2¼ 269, df¼ 7, p< .001.Table 3 shows the sampling categories and theirfrequencies. Total percentages for concurrent(71.7%) and sequential (28.3%) categories indi-cate that there is a direct correspondencebetween research designs and sampling designs.It should be noted that the research designs in the“Other” category that emerged in the process ofanalysis appeared to bemore sequential in nature,making this correspondence more meaningful. Itfollows that the concurrent procedures in thesampling categories occurred much more fre-quently than the sequential ones.

Analysis of the sampling designs also revealedthat the concurrent identical category was used inabout half of the sample articles (48.29%) and,thus, was the most common sampling design. Inthe same category, 17.07% of the samplingdesigns were multilevel. Concurrent paralleland nested designs were used in a very limitednumber of articles, 1.95% and 4.39%,respectively.


In the sequential category, on the other hand,almost all of the subcategories occurred with lowfrequencies. Having a frequency of 10.73% and9.76%, the sequential identical and multileveldesigns were more frequent than the other twosubcategories. The less frequent sampling designswere sequentially oriented parallel (1.95%) andnested (5.85%) designs.

Qualitative analysis of the data unveiled someintriguing aspects of sampling procedures used bythe researchers. The following extract illustratesan instance of using the identical samplingprocedure. In this type of sampling, one groupis selected for both the qualitative and quantita-tive phases of the study:

[18] ( . . .) participants were selected through oppor-tunity sampling, which, as defined in Martella,Nelson, and Marchand–Martella (1999), is a type ofconvenience sampling in that researchers use “mem-bers of the population who are willing to take part inthe research” (p. 125). ( . . .) The final sampleconsisted of 101 Bulgarian learners of English ( . . .)

(Charkova, 2007, pp. 378–379)

Most of the studies that used triangulation bymethods of analysis, particularly those utilizingqualitative and quantitative analysis of thesame corpus, used identical sampling in theirresearch:

[109] The MICASE corpus has been under compila-tion during the process of this study. This is one of thereasons for the multiple-method approach that wehave adopted. Another main motive was simplecuriosity concerning the suitability of an electronicspeech corpus for exploring the data in both

quantitative and qualitative ways, converging theangles on a single item.

(Lindemann & Mauranen, 2001, p. 463)

The following texts are examples of usingparallel samples. In parallel sampling designs, twoor more different groups, usually with similarfeatures, are selected from the same population:

[123] After having recorded, transcribed, analyzedand compared the three interactive lectures (Int A, IntB, and Int C) with the three non-interactive lectures(N1, N2, N3), the next step was to determine if thefindings could help to transform non-interactive intointeractive lecture discourse. To carry out this secondobjective, the three lecturers using the non-interactivemode agreed to attempt a more interactive style withanother group in the second semester of the sameacademic year. (Morell, 2004, p. 333)

[67] Prior to our intervention (phase 2), we carriedout an extensive investigation into the kinds ofdifficulties students in this population wereexperiencing using think-aloud procedures, with asample of students (N¼ 23) very similar to those whotook part in the intervention.

(Graham & Macaro, 2008, p. 757)

Nested samples were not used frequently in thedata sources. The examples below show how theauthors describe a nested sampling procedure inwhich the large sample participates in thequantitative phase and the small sample isselected from the initial sample for the qualitativephase of the study:

[139] The data were gathered during the finalexamination. There were 384 Thai students forquantitative data analyses, made up of 173 males

TABLE 3Frequencies of Mixed Methods Sampling Designs Used in the Articles

Journal

Concurrent Sampling Sequential Sampling

Identical Parallel Nested Multilevel Identical Parallel Nested Multilevel

F P% F P% F P% F P% F P% F P% F P% F P%

AL 19 65.52 0 0.00 1 3.45 3 10.34 5 17.24 0 0.00 0 0.00 1 3.45ESP 13 41.94 0 0.00 1 3.23 6 19.35 5 16.13 1 3.23 2 6.45 3 9.68LL 10 37.04 3 11.11 4 14.81 4 14.81 3 11.11 2 7.41 1 3.7 0 0.00LT 11 33.33 0 0.00 1 3.03 12 36.36 0 0.00 0 0.00 1 3.03 8 24.24MLJ 28 60.87 1 2.17 0 0.00 4 8.70 4 8.70 1 2.17 3 6.52 5 10.87TQ 13 43.33 0 0.00 2 6.67 4 13.13 4 13.13 0 0.00 5 16.67 2 6.67LTR 5 55.56 0 0.00 0 0.00 2 22.22 1 11.11 0 0.00 0 0.00 1 11.11Total 99 48.29 4 1.95 9 4.39 35 17.07 22 10.73 4 1.95 12 5.85 20 9.76

Note.Abbreviations: AL:Applied Linguistics; ESP: English for Specific Purposes; LL: Language Learning; LT: LanguageTesting; MLJ: TheModern Language Journal; TQ: TESOL Quarterly; LTR: Language Teaching Research. F and P standfor frequencies and percentages of mixed methods sampling designs in the sample.


(45%) and 211 females (55%). There were 75 highlysuccessful, 256 moderately successful and 53 unsuc-cessful test-takers. Eight (i.e., 4 highly successful and 4unsuccessful) test-takers were selected for retrospec-tive interviews. (Phakiti, 2003, p. 33)

[12] Two groups of participants were involved in thisstudy. ( . . .) To follow up on the quantitative results ofthe TA survey, 23 interviews were conducted with asubset of TAs from the same language departments assurveyed with the LMP. (Brandl, 2000, p. 357)

[97] From the 490 student–participants, 8 students (4females and 4 males) were selected to participate ininterviews. Stratified random sampling was used toselect these students in order to ensure thatmales andfemales were equally represented.

(Kissau & Wierzalis, 2008, p. 406)

Multilevel samples were more common thanparallel and nested samples. This type of samplinginvolves two or more different groups fromdifferent populations who take part in thequalitative and quantitative stages:

[30] We interviewed instructors shortly after theirESL courses were completed in eitherDecember 2000or early January 2001, once data on their students’performance on the prototype tasks (from theautumn of 2000) had been processed and madeavailable to us in CD-Rom form.

(Cumming et al., 2004, p. 114)

[148] The following four groups of participantsserved as readers for this study: 127 inexperiencedJapanese EFL student writers (IS), 128 experiencedJapanese EFL student writers (ES), 104 Japanese EFLteachers (JT), and 106 native English-speaking EFLteachers (NST) in Japan.

(Rinnert & Kobayashi, 2001, p. 191)

In most of the studies that used triangulation bysources (see Long, 2005), multilevel sampling wasused to collect and analyze data both qualitativelyand quantitatively.

Research Question 3

The third research question explored howgeneral conclusions or meta-inferences are devel-oped in applied linguistics mixed methodsresearch. As stated, this is only one aspect ofinference quality (Tashakkori & Teddlie, 2008); alack of operational definitions would, in any case,have made it nearly impossible to address all of itsaspects.

The analysis of the conclusion and discussionsections of the articles showed that independentsections on data integration and consolidation

were not presented in the studies3 and meta-inferences, in most of the cases, were notdeveloped based on qualitative and quantitativeinferences. However, in a good number of studies,general findings or interpretations were pre-sented for the sake of offering a summary.

[114] To sum up, the major findings of the presentstudy were ( . . .) (Louhiala–Salminen, 1996, p. 49)

[1] The initial study ( . . .) revealed shortcomings infour main areas which can be summarized as follows( . . .) (Allison et al., 1998, p. 212)

[169] The results that have had the greatest influenceon the content of the communication courses offeredat both institutions are discussed below: ( . . .)

(So–mui & Mead, 2000, p. 363)

[99] The results of the analysis of how the scripts ofthe role-play tasks affect conversational structuresuggest that rejections are rarely attempted, unlessthe script provides some opportunities for them.

(Klingner & Vaughn, 2000, p. 183)

Examples similar to the above extracts werefrequent in the data sources. This could demon-strate that in almost all of the studies thatcombined qualitative and quantitative researchgeneral conclusions and summaries were used todescribe the findings in an effective way. Theimportant point to be made is this: Althoughintegration of qualitative and quantitative find-ings did not appear as an independent section inthe articles, some authors did present overallconclusions that could be considered generalinferences:

[81] The distribution of these features shows that notall disciplines sanction the same degree of authorialpresence. Writers’ decisions are closely related to thesocial and epistemological practices of their disci-plines ( . . .) (Hyland, 2001, p. 224)

[187] Our conclusion is that uniformity of expressionin the business community is limited to the con-ventions imposed by the genre used.

(Vergaro, 2004, pp. 202–203)

[150] The findings of this “broad sweep” analysis arerather paradoxical. On one level themeetings appear“normal” and “orderly.” They also appear effective inthe sense that items on the agendas are covered andto time. (Rogerson–Revell, 2008, p. 356)

The above examples may indicate that generalconclusions or inferences were used by severalresearchers who attempted to combine qualitativeand quantitative methods in their studies. As amatter of fact, it does not appear that these


researchers formed meta-inferences by integrat-ing qualitative and quantitative inferences basedon data integration strategies to develop generalexplanations from integrated results. Rather, byand large, these general inferences were offeredby the researchers to enrich their discussions andconclusions or present informative summaries ofthe findings.

In addition to such general findings, closeexamination of the data led to the emergence ofgeneral inferences based on the data fromqualitative and quantitative components in anumber of studies:

[176] Taken together, the quantitative and qualita-tive data suggest that the teacher-coordinated inter-actions in the TLD groupmight have better equippedlearners for performance with the instructionalobject than the small-group learner-led interactionsin LLD. (Toth, 2008, pp. 267–268)

[43] Qualitative feedback from test-takers confirmswhat was revealed by the quantitative analyses;namely, the lack of any systematic relationshipbetween task difficulty and hypothesized task com-plexity, on the one hand, and actual test perfor-mance, on the other.

(Elder, Iwashita, & McNamara, 2002, p. 360)

[197] Quantitative and qualitative data indicated thatthe students valued the knowledge and the experi-ences gained through collaborative project work inauthentic settings. (Wilhelm, 1997, p. 533)

[134] The quantitative and qualitative findings of thefollow-up study provided empirical support by show-ing that experts’ domain specific written discoursediffered in a systematic way from that of otherteachers in our study.

(Olshtain & Kupferberg, 1998, p. 196)

[3] The qualitative and quantitative results of thisstudy show that taking part in a collaborative dialoguewith student teachers following INSET impactspositively on the professional development of partici-pating teachers. (Atay, 2004, p. 143)

As these examples may suggest, general infer-ences were developed based on data integration.Data integration in such studies involved featuressuch as data comparison (e.g., data sources [139],[171]), and data confirmation or complementar-ity (e.g., data sources [127], [134], [154], [176],[197]). Although none of the above studiesoffered separate sections on data integration orconsolidation, they developed their general con-clusions or inferences based on the resultsobtained from both qualitative and quantitativecomponents of the studies.

DISCUSSION

Use of Mixed Research Designs

Research designs that were explored in thepresent study corresponded well with thosepresented by Creswell et al. (2008) and sharedsimilar features with the designs discussed inGreene, Caracelli, and Graham (1989). Theanalysis of research designs revealed that themost commonly used design was the concurrentdesign. This finding is in line with Hanson et al.(2005) and similar to the findings ofmeta-analysisarticles on mixed methods research in the socialsciences, indicating that, generally, concurrentdesigns are more frequent than sequentialdesigns (Christ, 2007; Collins et al., 2006). Amongthe subcategories of concurrent and sequentialdesigns, triangulation in the concurrent categoryoutnumbered the other subcategories by a widemargin. About 66% of the articles used triangula-tion whereas other subcategories comprised asmall portion of the designs. It appears thatresearchers used triangulation designs as a self-sufficient strategy not as a subcategory of mixedmethods designs. However, parts of the findingssuggest that, although triangulation was notdefined in the sense in which it is used in mixedmethods research, it shared some functions withmixed methods triangulation (for a detaileddiscussion of the functions, see Hammersley,2008) such as validity checking (e.g., data sources[59], [163]), and seeking complementary information(e.g., data sources [10], [40], [122], [191]).Overall, triangulation seems to have been usedin such studies to enhance the design quality ofthe study.

The results of this study, especially the descrip-tive notes made on the nature and function ofmixing methods in the research articles, maypartially support the idea that the rationalebehind adopting a particular design could belinked to practical considerations in terms of whatthe studies were intended to investigate. It canthus be inferred that different mixed methodsdesigns were developed for different purposes (cf.Greene et al., 1989).

For example, triangulation was, in many cases,used for the purpose of providing supplementarydata in multilevel discourse analysis throughpresenting micro-genetic and macro-genetictreatment of the texts. It was also used in needsanalysis and program evaluation (e.g., data source[40]) to collect data from multiple sources orevaluate the data by multiple methods. Thisis particularly relevant to validation studies.


Triangulation, complementarity, and expansionare mixed method designs that were found byGreene et al. (1989) to be very useful forevaluation purposes.

A theme that emerged in the process ofqualitative analysis was the tendency in some ofthe journals or subdisciplines to use a particulardesign more frequently (cf. Alise & Teddlie,2010). In the present study, for instance, triangu-lation was found to be the most frequently useddesign in all of the journals. Moreover, articlespublished in ESP Journal and Language Testingseemed to use triangulation for needs assessment,program evaluation, and test validation. As this isnot a central finding of the present study, moreaccurate accounts of it would require furtherresearch.

Concurrent embedded designs were used insome cases to qualitatively explore the processeswithin an experimental design to present richerdescriptions of the processes (e.g., data sources[34], [105], [119]). Such designs were particular-ly used to investigate the process of secondlanguage acquisition (e.g., data sources [34],[105], from Language Learning). Concurrentembedded designs were found in articles pub-lished in The Modern Language Journal andLanguage Learning but not other journals.

Explaining the findings from the quantitativephase, sequential explanatory designs were main-ly used to enhance the explanatory power ofcoding and rating procedures (e.g., data source[11]), to supplement quantitative measurementof performances with detailed qualitative descrip-tions (e.g., data sources [12], [159]), and topresent qualitative evaluation of DIF items afterusing statistical analyses (e.g., data source [182]).

Sequential exploratory designs were utilized forthe purpose of expanding the scope of thequalitative phase and further investigating theproblem quantitatively by using a larger sample.Such designs would be used in evaluating and/ordeveloping measurement instruments like ques-tionnaires, tests, rating scales, inventories (e.g.,data sources [116], [165], [181]) and/or validat-ing measurement instruments (data sources [88],[131]). The use of mixed methods research indeveloping quantitative instruments has beentreated in Onwuegbuzie, Bustamante, & Nelson(2010) and Durham, Tan, & White (2011).

Measurement instruments can also be validatedthrough the use of sequential embedded designs(e.g., data source [61]). These designs would helpthe researcher to quantitatively examine cross-sections of the test development and validationprocess. Being used as a supplement to qualitative

ethnographical studies, sequential embeddeddesigns can be used to develop patterns in amore rigorous way (e.g., data source [108]).

The findings of the present study furtherrevealed that a number of the research designsused by applied linguistics researchers did notexist in the typology presented by Creswell et al.(2008), though similar combinations can befound in the literature. Nastasi et al. (2007), forinstance, proposed five mixed methods designs,two of which could be regarded as a combinationof sequential and concurrent designs. Theauthors identified a number of practical functionsfor these designs that proved to be similar to theones that emerged in the present study, includinginstrument development and validation, andprogram development and evaluation.

Finally, to reiterate an earlier point, the presentstudy explored all the articles in which qualitativeand quantitative components were found to be anecessary design component and produced datathat were considered to contribute to the findingsof the study. However, not all designs with thischaracteristic can be categorized as pure mixeddesigns (Hanson et al., 2005) or what Caracelli andGreene (1997) called mixed-method integrateddesigns. If, for example, both qualitative andquantitative methods are used in a study butnot interrelated and interconnected in their“conceptualization and execution” (Bergman,2008, p. 2) at all levels of the study, the studydesign would be categorized as a quasi-mixed design(Tashakkori & Creswell, 2007a; Teddlie &Tashakkori, 2009). It would be more appropriate,then, to classifymost of the designs investigated inthe present study as quasi-mixeddesigns. This is inline with Alise and Teddlie’s (2010) finding aboutthe high proportion of quasi-mixed designs acrossdisciplines.

Use of Mixed Sampling Designs

Results of the analysis of sampling designs werefairly similar to the findings of a previous studyconducted by Collins et al. (2007). The findingsfrom both studies indicate that concurrentidentical sampling is the most frequently usedsampling design. More particularly, concurrentdesigns were used in 71.7% of the articles in thepresent study; similarly, Collins et al. (2007)found that 66.1% of the sampling designs wereconcurrent. This finding, by contrast, is notconsistent with the results in Collins et al.(2006), in which sequential designs using multi-level samples were found to be the most frequent(40.5%).


Furthermore, identical sampling designs in theconcurrent category were the most prevalent(48.29%); this was consistent with the results fromCollins et al. (2007), who found a prevalence rateof 28.9% for this type of design. Despite suchsimilarities, there were also discrepancies betweenthe results of the two studies. For example, nestedsampling in the concurrent category was reportedby Collins et al. (2007) to be more frequent thanmultilevel designs, whereas the opposite was thecase in the present study. Finally, in both studies,parallel samples were found to be the leastfrequently used samples in the articles.

One major reason for the discrepancies in theresults of such studies might lie in the fact thatthey addressed samples of articles that weredifferent in terms of discipline, size of sample,research purpose, and rationale for mixing. Inthis sense, the choice of sampling design woulddiffer from project to project.

The reasons why concurrent designs withidentical samples were the most prevalent inapplied linguistics research may be related tofactors such as practicality and feasibility ofworking with identical samples. Identical samplesare convenient to work with and the researcherswould not have to put much time and energy intocollecting qualitative and quantitative data fromdifferent groups when they work with identicalsamples. This means that mixed data could beelicited from the same sample over a shorter spanof time. By contrast, working with nested, parallel,and multilevel samples could be more difficultand would require more time, especially whenthese samples are utilized in a sequential design.However, the choice of sampling design is directlybased on the research purpose and rationale formixing qualitative and quantitative methods.

Inference Quality

A lack of practical models for operationalizinginference quality in the present study was alimitation that led to the exploration of onlyone of its sub-components—i.e., integrative efficacy(Tashakkori & Teddlie, 2008). As the results maydemonstrate, only the barest mention was madeof the concept of inference quality in the studies.This finding reminds us of the results reported inO’Cathain, Murphy, & Nicholl (2007) and theirconcern about lack of visibility of the integrationquality in mixed methods studies. The reasonmight be that the concept has not yet beenintroduced appropriately to applied linguisticsresearch or that it has not been widely used inpractice. In a similar vein, studies that built the

interpretations on inferences from both qualita-tive and quantitative findings were limited innumber. This may be considered to be amethodological issue. As Creswell et al. (2008)argue, this situationmay be a consequence of onlymerging qualitative and quantitative data ratherthan developing a type of data which is built onthe different pieces of evidence from theseindependent strands. That is, presenting differ-ent types of data from a variety of sources anddeveloping general conclusions does not neces-sarily guarantee the validity of mixed researchand, therefore, cannot be an indicator for thepresence of interpretive rigor as an importantcomponent of inference quality. This said,mention should also be made that most of thearticles in this study did present general con-clusions and inferences or summaries of findings,although these were probably developed toenrich their discussion and conclusions.

Generally, this problem can be further dis-cussed in light of Tashakkori and Creswell’s(2007a) distinction between focus on methods(i.e., collecting and analyzing different types ofqualitative and quantitative data) and focus onmethodology (i.e., integration of the two approachesthroughout the study). Adopting a focus-on-methodology approach would enhance inferencequality by improving the quality of its maincomponents: design quality and interpretive rigor(see Tashakkori & Teddlie, 2008). On the whole,the main issue to address in terms of inferencequality in mixed methods research is “the degreeto which mixed methods researchers genuinelyintegrate their findings” (Bryman, 2007, p. 8). Inthis respect, the findings of the present study werein line with those of similar studies in the socialsciences as “substantial integration” of bothqualitative and quantitative approaches was foundto be “exercised very rarely” (Niglas, 2004, p. 98, ascited in Bryman, 2007, p. 9).

IMPLICATIONS OF THE STUDY

In applied linguistics, researchers often dealwith complex phenomena and systems with“adaptive,” “nonlinear,” and “dynamic” processesand multi-dimensional outcomes (Larsen–Free-man & Cameron, 2008, p. 251). To explore thesecomplicated processes and outcomes, we may, attimes, need to utilize both qualitative andquantitative methods (see Yin, 2006). The inte-gration of qualitative and quantitative methodswithin a systematically developed mixed methodsdesign may thus prove to be a useful tool forconducting research in applied linguistics.


Uncovering a number of functions and applica-tions for mixed methods research in appliedlinguistics is thus an important contribution ofthe present study, a possibility first projected byHashemi (2012).

Concurrent triangulation designs make use ofcomplementary data from both qualitative andquantitative strands in multilevel discourse analy-sis, genre analysis, needs analysis, and programevaluation. In such cases, the findings from onestrand would complement and/or cross-validatethe ones from the other.

Concurrent embedded designs can be used toprovide detailed qualitative analysis of the sub-systems that exist or processes that prevail withinan experiment. This type of design is useful forexploring language development in experimentsthat are designed to study acquisition of certain L2features. It may also be utilized to investigate thenature of learners’ performance and the strategiesthey use in the process of completing tasks.

Sequential explanatory designs may be appro-priate to the requirements of the studies inwhich qualitative explanation of the findingsfrom a quantitative phase would be illuminating.Using such designs, researchers may explore asubsample or a number of subsamples from thelarge sample(s) used in the quantitative phase(see Lieberman, 2005). This would enhance thequality of the study through analysis of extremecases, outliers, inconsistent components, incom-patible subsystems, and so on. This would createexplanatory power in coding and rating proce-dures in language assessment. This type of designwould also provide a range of possibilities forfurther analyzing and explaining performanceson quantitative measurement instruments, andqualitatively evaluating DIF items after usingstatistical analyses.

Sequential exploratory designs can be utilizedto add to the breadth of an investigation byquantitatively exploring the problems and ques-tions that may emerge from the qualitative phase.In an exploratory design, utilizing a quantitativephase can be considered as a solution to theproblem of generalizability of the qualitativefindings in the first phase. The qualitative phasemay also be used for piloting or evaluating aparticular scheme, method, or assessment tool.Such designs would be used in designing andvalidating measurement instruments and evalua-tion criteria such as language tests, surveys andquestionnaires, rating scales, and band descrip-tors (see Onwuegbuzie et al., 2010).

The use of sequential embedded designs wouldallow researchers to quantitatively examine cross-

sections of a qualitative process. For example, inlongitudinal studies of the developmental pro-cesses in language acquisition, researchers maybe interested in measuring learner perform-ances, learning outcomes, proficiency levels,and other quantifiable aspects of the process bydesigning sequential embedded studies. In testvalidation, too, this type of design would helpresearchers to quantitatively analyze cross-sec-tions of the test development and validationprocesses. In ethnographical studies, qualitativemixed research (see Denzin, 2012; Hesse–Biber, 2010) can be realized through the use ofsimilar sequential designs for enriching thedescriptions, developing patterns, and evaluatingmodels in a more rigorous way. This would beuseful in investigating learner discourse, teacherdiscourse, interaction patterns, learner identity,teacher identity, intercultural pragmatics, andother dimensions related to the social aspects oflanguage learning.

In sum, applied linguistics researchers can useinnovative mixed research and sampling designsbased on the research purpose and requirementsof the research context. It would be particularlyadvantageous to utilize contextualized mixedmethods data analytic strategies that are appro-priate for researching areas such as secondlanguage acquisition, language assessment, learn-er evaluation, teacher education, curriculumdesign, materials evaluation, materials develop-ment, discourse and pragmatics, languages forspecific purposes, and so forth. Also, researchersmay address the operationalization of evaluativecriteria for investigating the integration quality inapplied linguistics research.

We make this final point: High-quality integra-tion inmixedmethods research depends cruciallyon the design quality and quality of meta-inferences (Tashakkori & Teddlie, 2008), affirm-ing the importance of the integration of qualita-tive and quantitative components at all of thestages in a study to lead to interpretations that arebased on integrated results. This can help us notonly to further improve the quality of mixedmethods research in applied linguistics butcontribute in substantive ways to what we canand do know about language teaching andlearning.

ACKNOWLEDGMENT

We would like to thank Heidi Byrnes the MLJ editor,for her very helpful comments on the earlier drafts ofthe article.


NOTES

1 Although the term “rich description” is commonlyused in the context of ethnographical research, weborrow this general and inclusive term from Erickson(1986) to metaphorically refer to in-depth and detaileddescriptions of the content that are supplemented by“interpretive perspective” (see p. 150).

2 As one of the anonymous reviewers rightly men-tioned, the use of terminology cannot be the onlyindicator of the type of method or design used in thestudies. In fact, this was used only as a technique to findthe signaling elements in the content; we also attemptedto present comprehensive descriptions of the content toenrich our analysis.

3 For an excellent example of data integration seeJang et al. (2008).

REFERENCES

Alise, M. A., & Teddlie, C. (2010). A continuation of theparadigm wars? Prevalence rates of methodologi-cal approaches across the social/behavioral scien-ces. Journal of Mixed Methods Research, 4, 103–126.

Altman, D. G. (1991). Practical statistics for medicalresearch. London: Chapman and Hall.

Barkaoui, K. (2010). Do ESL raters’ evaluation criteriachange with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44, 31–57.

Bazeley, P. (2004). Issues in mixing qualitative andquantitative approaches to research. In R. Buber,J. Gadner, &L. Richards (Eds.),Applying qualitativemethods to marketing management research (pp. 141–156). Basingstoke, UK: Palgrave Macmillan.

Bazeley, P. (2009). Editorial: Integrating data analyses inmixed methods research. Journal of Mixed MethodsResearch, 3, 203–207.

Benson, P., Chik, A., Gao, X., Huang, J., & Wang, W.(2009). Qualitative research in language teachingand learning journals, 1997–2006. Modern Lan-guage Journal, 93, 79–90.

Bergman, M. M. (2008). Introduction: Whither mixedmethods? In M. M. Bergman (Ed.), Advances inmixed methods research (pp. 1–7). London: SAGE.

Bryman, A. (2007). Barriers to integrating quantitativeand qualitative research. Journal of Mixed MethodsResearch, 1, 8–22.

Bryman, A. (2008). Why do researchers integrate/combine/mesh/blend/mix/merge/fuse quanti-tative and qualitative research? In M. M. Bergman(Ed.), Advances in mixed methods research (pp. 87–100). London: SAGE.

Campbell, D. T. (1953). A study of leadership amongsubmarine officers. Columbus, OH: The Ohio StateUniversity.

Campbell, D. T., & Fiske, D. W. (1959). Convergent anddiscriminant validation by the multitrait–multi-method matrix. Psychological Bulletin, 56, 81–105.

Caracelli, V. J., & Greene, J. C. (1993). Data analysisstrategies for mixed-method evaluation designs.Educational Evaluation and Policy Analysis, 15, 195–207.

Caracelli, V. J., & Greene, J. C. (1997). Crafting mixed-method evaluation designs. In J. C. Greene & V. J.Caracelli (Eds.), Advances in mixed-method evalua-tion: The challenges and benefits of integrating diverseparadigms (New Directions for Evaluation, No. 74,pp. 19–32). San Francisco, CA: Jossey–Bass.

Chatterji, M. (2004). Evidence on “what works”:An argument for extended-term mixed-method(ETMM) evaluation designs. Educational Researcher,33, 3–13.

Christ, T. W. (2007). A recursive approach to mixedmethods research in a longitudinal study ofpostsecondary education disability support ser-vices. Journal of Mixed Methods Research, 1, 226–241.

Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G.(2006). Prevalence of mixed methods samplingdesigns in social science research. Evaluation andResearch in Education, 19, 83–101.

Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G.(2007). A mixed methods investigation of mixedmethods sampling designs in social and healthscience research. Journal of Mixed Methods Research,1, 267–294.

Creswell, J. W., & PlanoClark, V. L. (2007).Designing andconducting mixed methods research. Thousand Oaks,CA: SAGE.

Creswell, J. W., Plano Clark, V. L., & Garrett, A. L.(2008). Methodological issues in conductingmixed methods research designs. In M. M. Berg-man (Ed.), Advances in mixed methods research(pp. 66–83). London: SAGE.

Cumming, A. (1994). Alternatives in TESOL research:Descriptive, interpretive, and ideological orienta-tions. TESOL Quarterly, 28, 673–703.

Datta, L. (1997). A pragmatic basis for mixed-methoddesigns. In J. C. Greene & V. J. Caracelli (Eds.),Advances in mixed-method evaluation: The challengesand benefits of integrating diverse paradigms (NewDirections for Evaluation, No. 74, pp. 33–46). SanFrancisco, CA: Jossey–Bass.

Davis, K. A. (1995). Qualitative theory and methods inapplied linguistics research. TESOL Quarterly, 29,427–453.

Dellinger, A. B., & Leech, N. L. (2007). Toward a unifiedvalidation framework in mixed methods research.Journal of Mixed Methods Research, 1, 309–332.

Denscombe, M. (2008). Communities of practice: Aresearch paradigm for mixed methods approach.Journal of Mixed Methods Research, 2, 270–283.

Denzin, N. K. (1978). The research act: A theoreticalintroduction to sociological methods. New York:McGraw–Hill.

Denzin, N. K. (2012). Triangulation 2.0. Journal of MixedMethods Research, 6, 80–88.

Dornyei, Z. (2007). Research methods in applied linguistics:Quantitative, qualitative, and mixed methodologies.Oxford: Oxford University Press.


Durham, J., Tan, B.–K., & White, R. (2011). Utilizingmixed research methods to develop a quantitativeassessment tool: An example from explosiveremnants of a war clearance program. Journal ofMixed Methods Research, 5, 212–226.

Egbert, J. (2007). Quality analysis of journals in TESOLand applied linguistics. TESOL Quarterly, 41, 157–171.

Ellingson, L. L. (2009). Engaging crystallization inqualitative research. Thousand Oaks, CA: SAGE.

Ellingson, L. L. (2011). Analysis and representationacross the continuum. In N. K. Denzin & Y. S.Lincoln (Eds.), Handbook of qualitative research(4th ed., pp. 595–610). Thousand Oaks, CA:SAGE.

Erickson, F. (1986). Qualitative methods in research onteaching. In M. C. Wittrock (Ed.), Handbook ofresearch on teaching (pp. 119–161). New York:Collier–Macmillan.

Feilzer, M. Y. (2010). Doing mixed methods researchpragmatically: Implications for the rediscovery ofpragmatism as a research paradigm. Journal ofMixed Methods Research, 4, 6–16.

Freshwater, D. (2007). Reading mixed methodsresearch: Contexts for criticism. Journal of MixedMethods Research, 1, 134–146.

Fries, C. J. (2009). Bourdieu’s reflexive sociology as atheoretical basis for mixed methods research: Anapplication to complementary and alternativemedicine. Journal of Mixed Methods Research, 3,326–348.

Giddings, L., & Grant, B. (2007). A Trojan horse forpositivism? A critique of mixed methods research.Advances in Nursing Science, 30, 52–60.

Gorard, S. (with Taylor, C.). (2004).Combining methods ineducational and social research. Maidenhead, UK:Open University Press.

Greene, J. C. (2007). Mixed methods in social inquiry. SanFrancisco, CA: Jossey–Bass.

Greene, J. C. (2008). Is mixed methods social inquiry adistinctive methodology? Journal of Mixed MethodsResearch, 2, 7–22.

Greene, J. C., & Caracelli, V. J. (1997). Definingand describing the paradigm issue in mixed-method evaluation. In J. C. Greene & V. J.Caracelli (Eds.), Advances in mixed-method evalua-tion: The challenges and benefits of integratingdiverse paradigms (New Directions for Evaluation,No. 74, pp. 5–17). San Francisco, CA: Jossey–Bass.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989).Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluationand Policy Analysis, 11, 255–274.

Guest, G. (2013). Describing mixed methods research:An alternative to typologies. Journal of MixedMethods Research, 7, 141–151.

Hall, B., & Howard, H. (2008). A synergistic approach:Conducting mixed methods research with typo-logical and systemic design considerations. Journalof Mixed Methods Research, 2, 248–269.

Hammersley, M. (2008). Troubles with triangulation. InM. M. Bergman (Ed.), Advances in mixed methodsresearch (pp. 22–36). London: SAGE.

Hanson,W. E., Creswell, J. W., Plano Clark, V. L., Petska,K. S., & Creswell, J. D. (2005). Mixed methodsresearch designs in counseling psychology. Journalof Counseling Psychology, 52, 224–235.

Harrits, G. S. (2011).More thanmethod? A discussionofparadigm differences within mixed methodsresearch. Journal of Mixed Methods Research, 5,150–166.

Hashemi, M. R. (2012). Reflections on mixing methodsin applied linguistics research. Applied Linguistics,33, 206–212.

Hesse–Biber, S. N. (2010).Mixed methods research: Mergingtheory with practice. New York: Guilford Press.

Jang, E. E., McDougall, D. E., Pollen, D., Herbert, M., &Russell, P. (2008). Integrativemixedmethods dataanalytic strategies in research on school successin challenging circumstances. Journal of MixedMethods Research, 2, 221–247.

Johnson, R. B. (2008). Living with tensions: Thedialectic approach. Journal of Mixed MethodsResearch, 2, 203–207.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixedmethods research: A research paradigm whosetime has come. Educational Researcher, 33, 14–26.

Jung, U. O. H. (2004). Paris in London revisited or theforeign language teacher’s topmost journals.System, 32, 357–361.

Kim, Y.–H. (2009). An investigation into native and non-native teachers’ judgments of oral English perfor-mance: A mixed methods approach. LanguageTesting, 26, 187–217.

Kissau, S., & Wierzalis, E. (2008). Gender identity andhomophobia: The impact on adolescent malesstudying French.Modern Language Journal, 92, 402–413.

Kuhn, T. S. (1962). The structure of scientific revolutions.Chicago: University of Chicago Press.

Lamb, M. (2007). The impact of school on EFL learningmotivation: An Indonesian case study. TESOLQuarterly, 41, 757–780.

Larsen–Freeman, D., & Cameron, L. (2008). Complexsystems and applied linguistics. Oxford: OxfordUniversity Press.

Lazaraton, A. (1995). Qualitative research in appliedlinguistics: A progress report. TESOL Quarterly, 29,445–472.

Lazaraton, A. (2000). Current trends in researchmethodology and statistics in applied linguistics.TESOL Quarterly, 34, 175–181.

Lazaraton, A. (2002). Qualitative and quantitativeapproaches to discourse analysis. Annual Reviewof Applied Linguistics, 22, 32–51.

Lazaraton, A. (2003). Evaluative criteria for qualitativeresearch in applied linguistics: Whose criteria andwhose research? Modern Language Journal, 87,1–12.

Lazaraton, A. (2005). Quantitative researchmethods. InE. Hinkel (Ed.), Handbook of research in second


language teaching and learning (pp. 209–224).Mahwah, NJ: Lawrence Erlbaum.

Lee, Y., & Greene, J. (2007). The predictive validity of anESL placement test: A mixed methods approach.Journal of Mixed Methods Research, 1, 366–389.

Leeuw, E., & Hox, J. (2008). Mixing data collectionmethods: Lessons from social survey research. InM. M. Bergman (Ed.), Advances in mixed methodsresearch (pp. 138–149). London: SAGE.

Lieberman, E. S. (2005). Nested analysis as a mixed-method strategy for comparative research. Ameri-can Political Science Review, 99, 435–452.

Long, M. H. (2005). Methodological issues in learnerneeds analysis. In M. H. Long (Ed.), Secondlanguage needs analysis (pp. 19–76). Cambridge:Cambridge University Press.

Mackey, A. (2006). Feedback, noticing and instructedsecond language learning. Applied Linguistics, 27,405–430.

Martella, R. C., Nelson, R., & Marchand–Martella, N.(1999). Research methods: Learning to become a criticalresearch consumer. Needham Heights, MA: Allyn &Bacon.

McConney, A., Rudd, A., & Ayres, R. (2002). Getting tothe bottom line: A method for synthesizingfindings within mixed-method program evalua-tions. American Journal of Evaluation, 23, 121–140.

Mertens,D.M. (2007). Transformative paradigm:Mixedmethods and social justice. Journal ofMixedMethodsResearch, 1, 212–225.

Mertens, D. M. (2010). Philosophy in mixed methodsteaching: The transformative paradigm as illustra-tion. International Journal of Multiple ResearchApproaches, 4, 9–18.

Morgan, D. L. (2007). Paradigms lost and pragmatismregained:Methodological implications of combin-ing qualitative and quantitativemethods. Journal ofMixed Methods Research, 1, 48–76.

Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G.,Varjas, K., & Jayasena, A. (2007). Mixed methodsin intervention research: Theory to adaptation.Journal of Mixed Methods Research, 1, 164–182.

Newman, I., & Benz, C. R. (1998). Qualitative–quantita-tive research methodology: Exploring the interactivecontinuum. Carbondale, IL: Southern IllinoisUniversity Press.

Niglas, K. (2004). The combined use of qualitative andquantitative methods in educational research.(Unpublished doctoral dissertation). TallinnPedagogical University, Tallinn, Estonia.

Niglas, K., Kaipainen, M., & Kippar, J. (2008). Multi-perspective exploration as a tool for mixedmethods research. In M. M. Bergman (Ed.),Advances in mixed methods research (pp. 172–188).London: SAGE.

Norton, B. (1995). The theory of methodology inqualitative research.TESOLQuarterly, 29, 569–576.

O’Cathain, A., Murphy, E., & Nicholl, J. (2007).Integration and publications as indicators of“yield” from mixed methods studies. Journal ofMixed Methods Research, 1, 147–163.

Onwuegbuzie, A. J., Bustamante, R. M., & Nelson, J. A.(2010). Mixed research as a tool for developingquantitative instruments. Journal of Mixed MethodsResearch, 4, 56–78.

Onwuegbuzie, A. J., & Dickinson, W. B. (2008). Mixedmethods analysis and information visualization:Graphical display for effective communication ofresearch results. The Qualitative Report, 13, 204–225.

Onwuegbuzie, A. J., & Johnson, R. B. (2006). Thevalidity issue in mixed research. Research in theSchools, 13, 48–63.

Onwuegbuzie, A. J., & Leech, N. L. (2006). Linkingresearch questions to mixed methods dataanalytic procedures. The Qualitative Report, 11,474–498.

Patton, M. Q. (2002). Qualitative evaluation and researchmethods (3rd ed.) Thousand Oaks, CA: SAGE.

Plonsky, L., & Gass, S. (2011). Quantitative researchmethods, study quality, and outcomes: The case ofinteraction research. Language Learning, 61, 325–366.

Pritchard, R. M. O., & Nasr, A. (2004). Improvingreading performance among Egyptian engineer-ing students: Principles and practice. English forSpecific Purposes, 23, 425–445.

Ridenour, C. S., & Newman, I. (2008). Mixed methodsresearch: Exploring the interactive continuum. Carbon-dale, IL: Southern Illinois University Press.

Saif, S. (2006). Aiming for positive washback: A casestudy of international teaching assistants. Lan-guage Testing, 23, 1–34.

Sale, J. E. M., Lohfeld, L. H., & Brazil, K. (2002).Revisiting the quantitative–qualitative debate:Implications for mixed-methods research. Quality& Quantity, 36, 43–53.

Sasaki,M. (2004). Amultiple-data analysis of the 3.5-yeardevelopment of EFL student writers. LanguageLearning, 54, 525–582.

Sechrest, L., & Sidana, S. (1995). Quantitative andqualitative methods: Is there an alternative?Evaluation and Program Planning, 18, 77–87.

Tan, M. (2011). Mathematics and science teachers’beliefs and practices regarding the teaching oflanguage in content learning. Language TeachingResearch, 15, 325–342.

Tarrow, S. (2004). Bridging the quantitative–qualitativedivide. In H. E. Brady & D. Collier (Eds.),Rethinking social inquiry: Diverse tools, shared stand-ards (pp. 171–179). Lanham, MD: Rowman &Littlefield.

Tashakkori, A., & Creswell, J. W. (2007a). Editorial: Thenew era ofmixedmethods. Journal ofMixedMethodsResearch, 1, 3–7.

Tashakkori, A., & Creswell, J. W. (2007b). Editorial:Exploring the nature of research questions inmixed methods research. Journal of Mixed MethodsResearch, 1, 207–211.

Tashakkori, A., & Teddlie, C. (1998). Mixed methodology:Combining qualitative and quantitative approaches.Thousand Oaks, CA: SAGE.


Tashakkori, A., & Teddlie, C. (2003). The past andfuture of mixed methods research: From datatriangulation to mixed model designs. In A.Tashakkori & C. Teddlie (Eds.),Handbook of mixedmethods in social and behavioral research (pp. 671–702). Thousand Oaks, CA: SAGE.

Tashakkori, A., & Teddlie, C. (2008). Quality ofinferences in mixed methods research. In M. M.Bergman (Ed.), Advances in mixed methods research(pp. 101–119). London: SAGE.

Tashakkori, A., & Teddlie, C. (2010). Editorial: Puttingthe human back in the “human research method-ology”: The researcher inmixedmethods research.Journal of Mixed Methods Research, 4, 271–277.

Teddlie, C., & Tashakkori, A. (2009). The foundations ofmixed methods research: Integrating quantitative andqualitative techniques in the social and behavioralsciences (2nd ed.). Thousand Oaks, CA: SAGE.

Teddlie, C., & Yu, F. (2007).Mixedmethods sampling: Atypology with examples. Journal of Mixed MethodsResearch, 1, 77–100.

Vavrus, F. (2002). Postcoloniality and English: Ex-ploring language policy and the politics of

development in Tanzania. TESOL Quarterly, 36,373–397.

Weber, M., & Campbell, C. M. (2004). In otherprofessional journals. Modern Language Journal,88, 457–466.

Wilhelm, K. H. (1997). Sometimes kicking and scream-ing: Language teachers-in-training react to acollaborative learning model. Modern LanguageJournal, 81, 527–542.

Woolley, C. M. (2009). Meeting the mixed methodschallenge of integration in a sociological study ofstructure and agency. Journal of Mixed MethodsResearch, 3, 7–25.

Yin, R. K. (2006). Mixed methods research: Are themethods genuinely integrated or merely parallel?Research in the Schools, 13, 41–47.

Yu, M.–C. (2008). Teaching and learning sociolinguisticskills in university EFL classes in Taiwan. TESOLQuarterly, 42, 31–53.

Zhang, Y. (2011). Judgments of oral proficiency by non-native and native English speaking teacher raters:Competing or complementary constructs? Lan-guage Testing, 28, 31–50.

APPENDIX A: Data Sources

[1] Allison, D., Cooley, L., Lewkowicz, J., & Nunan, D. (1998). Dissertation writing in action: The development of adissertation writing support program for ESL graduate research students. English for Specific Purposes, 17, 199–217.

[2] Arnold, J. (2000). Seeing through listening comprehension exam anxiety. TESOL Quarterly, 34, 777–786.[3] Atay, D. (2004). Collaborative dialogue with student teachers as a follow-up to teacher in-service education and

training. Language Teaching Research, 8, 143–162.[4] Ayoun, D. (2004). The effectiveness of written recasts in the second language acquisition aspectual distinctions in

French: A follow-up study. Modern Language Journal, 88, 31–55.[5] Bateman, B. E. (2002). Promoting openness toward culture learning: Ethnographic interviews for students of Spanish.

Modern Language Journal, 86, 318–331.[6] Beeching, K. (1997). French for specific purposes: The case for spoken corpora. Applied Linguistics, 18, 374–394.[7] Biber, D., & Barbieri, F. (2007). Lexical bundles in university spoken and written registers. English for Specific Purposes,

26, 263–286.[8] Bongartz, C., & Schneider, M. L. (2003). Linguistic development in social contexts: A study of two brothers learning

German. Modern Language Journal, 87, 13–37.[9] Borg, S., & Burns, A. (2008). Integrating grammar in adult TESOL classrooms. Applied Linguistics, 29, 456–482.[10] Bosher, S. (1997). Language and cultural identity: A study of Hmong students at the postsecondary level. TESOL

Quarterly, 31, 593–603.[11] Bowles, H. (2006). Bridging the gap between conversation analysis and ESP–An applied study of the opening

sequences of NS and NNS service telephone calls. English for Specific Purposes, 25, 332–357.[12] Brandl, K. K. (2000). Foreign language TA’s perceptions of training components: Do we know how they like to be

trained? Modern Language Journal, 84, 355–371.[13] Brindley, G., & Slatyer, H. (2002). Exploring task difficulty in ESL listening assessment. Language Testing, 19, 369–

394.[14] Caldas, S. J. (2008). Changing bilingual self-perceptions from early adolescence to early adulthood: Empirical

evidence from a mixed-methods case study. Applied Linguistics, 29, 290–311.[15] Chamot, A. U., & El–Dinary, P. B. (1999). Children’s learning strategies in language immersion classrooms. Modern

Language Journal, 83, 319–338.[16] Chang, A. C.–S., & Read, J. (2006). The effects of listening support on the listening performance of EFL learners.

TESOL Quarterly, 40, 375–397.[17] Chang, Y.–F. (2006). On the use of the immediate recall task as a measure of second language reading

comprehension. Language Testing, 23, 520–543.[18] Charkova, K. D. (2007). A language without borders: English slang and Bulgarian learners of English. Language

Learning, 57, 369–416.


[19] Charles, M. (2006). The construction of stance in reporting clauses: A cross-disciplinary study of theses. AppliedLinguistics, 27, 492–518.

[20] Chen, H.–C., & Graves, M. F. (1995). Effects of previewing and providing background knowledge on Taiwanesecollege students’ comprehension of American short stories. TESOL Quarterly, 29, 663–686.

[21] Cheng, W., & Warren, M. (1999). Inexplicitness: What is it and should we be teaching it? Applied Linguistics, 20, 293–315.

[22] Cheng, W., & Warren, M. (2005). Peer assessment of language proficiency. Language Testing, 22, 93–121.[23] Chimbganda, A. B. (2000). Communication strategies used in the writing of answers in biology by ESL first year

science students of the University of Botswana. English for Specific Purposes, 19, 305–329.[24] Choi, I.–C., Kim, K. S., & Boo, J. (2003). Comparability of a paper-based language test and a computer-based language

test. Language Testing, 20, 295–320.[25] Cohen, A. D., & Shively, R. A. (2007). Acquisition of requests and apologies in Spanish and French: Impact of study

abroad and strategy-building intervention. Modern Language Journal, 91, 189–212.[26] Cohen, A. D., &Upton, A. (2007). “I want to go back to the text”: Response strategies on the reading subtest of the new

TOEFL. Language Testing, 24, 209–250.[27] Crawford Camiciottoli, B. (2005). Adjusting a business lecture for an international audience: A case study. English for

Specific Purposes, 24, 183–199.[28] Cruz Cabanillas, I., Martinez, C. T., Prados, M. D., & Redondo, E. C. (2007). English loan words in Spanish computer

language. English for Specific Purposes, 26, 52–78.[29] Cullen, R., &Kuo, I.–C. (2007). Spoken grammar andELT coursematerials: Amissing link?TESOLQuarterly, 41, 361–

386.[30] Cumming, A., Grant, L., Mulcahy–Ernt, P., & Powers, D. E. (2004). A teacher-verification study of speaking and

writing prototype tasks for a new TOEFL. Language Testing, 21, 107–145.[31] Cutting, J. E. (1999). The grammar of the in-group code. Applied Linguistics, 20, 179–202.[32] Dalton–Puffer, C., & Nikula. T. (2006). Pragmatics of content-based instruction: Teacher and student directives in

Finnish and Austrian classrooms. Applied Linguistics, 27, 241–267.[33] David, G. (2007). Investigating the performance of alternative types of grammar items. Language Testing, 24, 65–97.[34] Davis, J. M. (2007). Resistance to L2 pragmatics in the Australian ESL context. Language Learning, 57, 611–649.[35] de la Fuente, M. J. (2006). Classroom L2 vocabulary acquisition: Investigating the role of pedagogical tasks and form-

focused instruction. Language Teaching Research, 10, 263–295.[36] Derwing, T. M., Munro, M. J., & Thomson, R. I. (2008). A longitudinal study of ESL learners’ fluency and

comprehensibility development. Applied Linguistics, 29, 359–380.[37] Dewaele, J.–M. (1998). Lexical inventions: French interlanguage as L2 versus L3. Applied Linguistics 19, 471–490.[38] Dewaele, J.–M., & Pavlenko, A. (2002). Emotion vocabulary in interlanguage. Language Learning, 52, 263–322.[39] Ding, H. (2007). Genre analysis of personal statements: Analysis of moves in application essays to medical and dental

schools. English for Specific Purposes, 26, 368–392.[40] Donato, R., Antonek, J. L., & Tucker, G. R. (1996). Monitoring and assessing a Japanese FLES program: Ambiance

and achievement. Language Learning, 46, 497–528.[41] Dong, Y. R. (1998). Non-native graduate students’ thesis/dissertation writing in science: Self-reports by students and

their advisors from two U.S. institutions. English for Specific Purposes, 17, 369–390.[42] Elder, C. (1996). The effect of language background on “foreign” language test performance: The case of Chinese,

Italian, and Modern Greek. Language Learning, 46, 233–282.[43] Elder, C., Iwashita, N., & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: What does the test-

taker have to offer? Language Testing, 19, 347–368.[44] Ellis, R., Basturkmen, H., & Loewen, S. (2001). Preemptive focus on form in the ESL classroom. TESOL Quarterly, 35,

407–432.[45] Ellis, R., & Yuan, F. (2003). The effects of pre-task planning and online planning on fluency, complexity and accuracy

in L2 monologic oral production. Applied Linguistics, 24, 1–27.[46] Evans, M., & Fisher, L. (2005). Measuring gains in pupils’ foreign language competence as a result of participation in

a school exchange visit: The case of Y9 pupils at three comprehensive schools in theUK. Language Teaching Research, 9,173–192.

[47] Fecteau, M. (1999). First- and second-language reading comprehension of literary texts.Modern Language Journal, 83,475–493.

[48] Ferguson, G. (2001). If you pop over there: A corpus-based study of conditionals in medical discourse. English forSpecific Purposes, 20, 61–82.

[49] Ferris, D. (1995). Student reactions to teacher response in multiple-draft composition classrooms. TESOL Quarterly,29, 33–53.

[50] Ferris, D., & Tagg, T. (1996). Academic listening/speaking tasks for ESL students: Problems, suggestions, andimplications. TESOL Quarterly, 30, 297–320.

[51] Ferris, D., & Tagg, T. (1996). Academic oral communication needs of EAP learners: What subject-matter instructorsactually require. TESOL Quarterly, 30, 31–58.

[52] Flowerdew, L. (2003). A combined corpus and systemic–functional analysis of the problem–solution pattern in astudent and professional corpus of technical writing. TESOL Quarterly, 37, 489–511.

[53] Folse, K. S. (2006). The effect of type of written exercise on L2 vocabulary retention. TESOL Quarterly, 40, 273–293.


[54] Fortanet, I. (2004). The use of “we” in university lectures: Reference and function. English for Specific Purposes, 23,45–66.

[55] Foster, P., & Ohta, A. S. (2005). Negotiation for meaning and peer assistance in second language classrooms. AppliedLinguistics, 26, 402–430.

[56] Fox, J. (2004). Test decisions over time: Tracking validity. Language Testing, 21, 437–465.[57] Frank, R. A. (2000). Medical communication: Non-native English speaking patients and native English speaking

professionals. English for Specific Purposes, 19, 31–62[58] Frazier, S. (2003). A corpus analysis of would-clauses without adjacent if-clauses. TESOL Quarterly, 37, 443–466.[59] Fulcher, G. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction.

Language Testing, 13, 208–238.[60] Fulcher, G. (1996). Testing tasks: Issues in task design and the group oral. Language Testing, 13, 23–51.[61] Fulcher, G. (1997). An English language placement test: Issues in reliability and validity. Language Testing, 14, 113–

138.[62] Fung, L., & Carter, R. (2007). Discourse markers and spoken English: Native and learner use in pedagogic settings.

Applied Linguistics, 28, 410–439.[63] Garrett, P., & Shortall, T. (2002). Learners’ evaluations of teacher-fronted and student-centred classroom activities.

Language Teaching Research, 6, 25–57.[64] Gatbonton, E. (1999). Investigating experienced ESL teachers’ pedagogical knowledge.Modern Language Journal, 83,

35–50.[65] Gosden, H. (1995). Success in research article writing and revision: A social-constructionist perspective. English for

Specific Purposes, 14, 37–57.[66] Graham, S. J. (2004). Giving up on modern foreign languages? Students’ perceptions of learning French. Modern

Language Journal, 88, 171–191.[67] Graham, S., &Macaro, E. (2008). Strategy instruction in listening for lower-intermediate learners of French.Language

Learning, 58, 747–783.[68] Green, C. F., Christopher, E. R., Lam, J., &Mei, K. (2000). The incidence and effects on coherence ofmarked themes

in interlanguage texts: A corpus-based enquiry. English for Specific Purposes, 19, 99–113.[69] Hardy, I. M., & Moore, J. L. (2004). Foreign language students’ conversational negotiations in different task

environments. Applied Linguistics, 25, 340–370.[70] Harwood, N. (2005). I hoped to counteract the memory problem, but I made no impact whatsoever: Discussing

methods in computing science using I. English for Specific Purposes, 24, 243–267.[71] Harwood, N. (2005). “We do not seem to have a theory . . . the theory I present here attempts to fill this gap”: Inclusive

and exclusive pronouns in academic writing. Applied Linguistics, 26, 343–375.[72] He, L., & Dai, Y. (2006). A corpus-based investigation into the validity of the CET–SET group discussion. Language

Testing, 23, 370–401.[73] Hedgcock, J., & Lefkowitz, N. (1996). Some input on input: Two analyses of student response to expert feedback in L2

writing. Modern Language Journal, 80, 287–308.[74] Heining–Boynton, A., &Haitema, T. (2007). A ten-year chronicle of student attitudes toward foreign language in the

elementary school. Modern Language Journal, 91, 149–168.[75] Herron, C., Corrie, C., Cole, S. P., & Dubreil, S. (1999). The effectiveness of a video-based curriculum in teaching

culture. Modern Language Journal, 83, 518–533.[76] Hewings, M., & Hewings, A. (2002). “It is interesting to note that . . . ”: A comparative study of anticipatory ‘it’ in

student and published writing. English for Specific Purposes, 20, 367–383.[77] Hinkel, E. (2003). Simplicity without elegance: Features of sentences in L1 and L2 academic texts. TESOL Quarterly,

37, 275–301.[78] Howarth, P. (1998). Phraseology and second language proficiency. Applied Linguistics, 19, 24–44.[79] Hu, G. (2005). Contextual influences on instructional practices: A Chinese case for an ecological approach to ELT.

TESOL Quarterly, 39, 635–660.[80] Hyland, K. (1999). Academic attribution: Citation and the construction of disciplinary knowledge. Applied Linguistics,

20, 341–367.[81] Hyland, K. (2001). Humble servants of the discipline? Self-mention in research articles. English for Specific Purposes, 20,

207–226.[82] Hyland, K. (2004). Graduates’ gratitude: The generic structure of dissertation acknowledgements. English for Specific

Purposes, 23, 303–324.[83] Hyland, K., & Tse, P. (2007). Is there an “academic vocabulary”? TESOL Quarterly, 41, 235–253.[84] Iwashita, N., &McNamara, T. (2001). Can we predict task difficulty in an oral proficiency test? Exploring the potential

of an information-processing approach to task design. Language Learning, 51, 401–436.[85] Izumi, S., & Bigelow, M. (2000). Does output promote noticing and second language acquisition? TESOL Quarterly,

34, 239–278.[86] Jackson, J. (2004). Case-based teaching in a bilingual context: Perceptions of business faculty in Hong Kong. English

for Specific Purposes, 23, 213–232.[87] Jennings, M., Fox, J., Graves, B., & Shohamy, E. (1999). The test-takers’ choice: An investigation of the effect of topic

on language-test performance. Language Testing, 16, 426–456.[88] Jianda, L. (2007). Developing a pragmatics test for Chinese EFL learners. Language Testing, 24, 391–415.


[89] Jones, F. R. (1998). Self-instruction and success: A learner-profile study. Applied Linguistics, 19, 378–406.[90] Jung, E. H. (2004). Topic and subject prominence in interlanguage development. Language Learning, 54, 713–738.[91] Kamhi–Stein, L. D. (2000). Looking to the future of TESOL teacher education:Web-based bulletin board discussions

in a methods course. TESOL Quarterly, 34, 423–455.[92] Katz, S., & Watzinger–Tharp, J. (2005). Toward an understanding of the role of applied linguistics in foreign

language departments. Modern Language Journal, 89, 490–502.[93] Kempe, V. (2001). The role of diminutives in the acquisition of Russian gender: Can elements of child-directed

speech aid in learning morphology? Language Learning, 51, 221–256.[94] Kern, R. G. (1995). Restructuring classroom interaction with networked computers: Effects on quantity and

characteristics of language production. Modern Language Journal, 79, 457–476.[95] Kim, M. (2001). Detecting DIF across the different language groups in a speaking test. Language Testing, 18, 89–114.[96] Kim, Y. (2008). The contribution of collaborative and individual tasks to the acquisition of L2 vocabulary. Modern

Language Journal, 92, 114–130.[97] Kissau, S., & Wierzalis, E. (2008). Gender identity and homophobia: The impact on adolescent males studying

French. Modern Language Journal, 92, 402–413.[98] Klingner, J. K., & Vaughn, S. (2000). The helping behaviors of fifth graders while using collaborative strategic reading

during ESL content classes. TESOL Quarterly, 34, 69–98.[99] Kormos, J. (1999). Simulating conversations in oral proficiency assessment: A conversation analysis of role plays and

non-scripted interviews in language exams. Language Testing, 16, 163–188.[100] Koteyko, N., Nerlich, B., Crawford, P., & Wright, N. (2008). “Not rocket science” or “no silver bullet”? Media and

government discourses about MRSA and cleanliness. Applied Linguistics, 29, 223–243.[101] Kuo, C. (1999). The use of personal pronouns: Role relationships in scientific journal articles. English for Specific

Purposes, 18, 121–138.[102] Lamb, M. (2007). The impact of school on EFL learningmotivation: An Indonesian case study. TESOL Quarterly, 41,

757–780.[103] Larsen–Freeman, D. (2006).The emergence of complexity, fluency, and accuracy in the oral and written production

of five Chinese learners of English. Applied Linguistics, 27, 590–619.[104] Lefkowitz, N., & Hedgcock, J. (2002). Sound barriers: Influences of social prestige, peer pressure and teacher (dis)

approval on FL oral performance. Language Teaching Research, 6, 223–244.[105] Leow, R. P. (1997). Attention, awareness, and foreign language behavior. Language Learning, 47, 467–505.[106] Leow, R. P., Hsieh, H.–C., & Moreno, N. (2008). Attention to form and meaning revisited. Language Learning, 58,

665–695.[107] Lepetit, D., & Cichocki, W. (2002). Teaching languages to future health professionals: A needs assessment study.

Modern Language Journal, 86, 384–396.[108] Liebscher, G., & Dailey–O’Cain, J. (2003). Conversational repair as a role-defining mechanism in classroom

interaction. Modern Language Journal, 87, 375–390.[109] Lindemann, S., & Mauranen, A. (2001). “It’s just real messy”: The occurrence and function of just in a corpus of

academic speech. English for Specific Purposes, 20, 459–475.[110] Liu, D., Ahn, G.–S., Baek, K.–S., & Han, N.–O. (2004). South Korean high school English teachers’ code switching:

Questions and challenges in the drive for maximal use of English in teaching. TESOL Quarterly, 38, 605–638.[111] Lockhart, C., & Ng, P. (1995). Analyzing talk in ESL peer response groups: Stances, functions, and content.

Language Learning, 45, 605–655.[112] Loewen, S. (2003). Variation in the frequency and characteristics of incidental focus on form. Language Teaching

Research, 7, 315–345.[113] Long, M. H., Inagaki, S., & Ortega, L. (1998). The role of implicit negative feedback in SLA: Models and recasts in

Japanese and Spanish. Modern Language Journal, 82, 357–371.[114] Louhiala–Salminen, L. (1996). The business communication classroom vs. reality: What should we teach today?

English for Specific Purposes, 15, 37–51.[115] Louhiala–Salminen, L., Charles, M., & Kankaanranta, A. (2005). English as a lingua franca in Nordic corporate

mergers: Two case companies. English for Specific Purposes, 24, 401–421.[116] Luxia, Q. (2005). Stakeholders’ conflicting aims undermine the washback function of a high-stakes test. Language

Testing, 22, 142–173.[117] Mackey, A. (2006). Feedback, noticing and instructed second language learning. Applied Linguistics, 27, 405–430.[118] Mackey, A., Kanganas, A. P., &Oliver, R. (2007). Task familiarity and interactional feedback in child ESL classrooms.

TESOL Quarterly, 41, 285–312.[119] Mackey, A., & Philp, J. (1998). Conversational interaction and second language development: Recasts, responses,

and red herrings? Modern Language Journal, 82, 338–356.[120] Manchon, R.M., &Roca deLarios, J. (2007). On the temporal nature of planning in L1 andL2 composing.Language

Learning, 57, 549–593.[121] Mantle–Bromley, C. (1995). Positive attitudes and realistic beliefs: Links to proficiency.Modern Language Journal, 79,

372–386.[122] McGroarty, M. E., & Zhue, W. (1997). Triangulation in classroom research: A study of peer revision. Language

Learning, 47, 1–43.[123] Morell, T. (2004). Interactive lecture discourse for university EFL students. English for Specific Purposes, 23, 325–338.


[124] Mori, Y. (2003). The roles of contexts and word morphology in learning new kanji words.Modern Language Journal,87, 404–420.

[125] Mullock, B. (2006). The pedagogical knowledge base of four TESOL teachers.Modern Language Journal, 90, 48–66.[126] Mustafa, Z. (1995). The effect of genre awareness on linguistic awareness. English for Specific Purposes, 14, 247–256.[127] Nakahama, Y., Tyler, A., & van Lier, L. (2001). Negotiation of meaning in conversational and information gap

activities: A comparative discourse analysis. TESOL Quarterly, 35, 377–405.[128] Nakatani, Y. (2005). The effects of awareness-raising training on oral communication strategy use.Modern Language

Journal, 89, 76–91.[129] Nemtchinova, E. (2005). Host teachers’ evaluations of nonnative-English-speaking teacher trainees: A perspective

from the classroom. TESOL Quarterly, 39, 235–261.[130] New, E. (1999). Computer-aided writing in French as a foreign language: A qualitative and quantitative look at the

process of revision. Modern Language Journal, 83, 80–97.[131] North, B., & Schneider, G. (1998). Scaling descriptors for language proficiency scales.Language Testing, 15, 217–263.[132] Okamura, A. (1995). Teachers’ and nonteachers’ perception of elementary learners’ spoken Japanese. Modern

Language Journal, 79, 29–40.[133] Oliver, R. (2000). Age differences in negotiation and feedback in classroom and pairwork. Language Learning, 50,

119–151.[134] Olshtain, E., &Kupferberg, I. (1998). Reflective–narrative discourse of FL teachers exhibits professional knowledge.

Language Teaching Research, 2, 185–202.[135] Papajohn, D. (1999). The effect of topic variation in performance testing: The case of the chemistry TEACH test for

international teaching assistants. Language Testing, 16, 52–81.[136] Paribakht, T. S. (2005). The influence of first language lexicalization on second language lexical inferencing: A

study of Farsi-speaking learners of English as a foreign language. Language Learning, 55, 701–748.[137] Pavlenko, A., & Driagina, V. (2007). Russian emotion vocabulary in American learners’ narratives.Modern Language

Journal, 91, 213–234.[138] Pearson, L. (2006). Patterns of development in Spanish L2 pragmatic acquisition: An analysis of novice learners’

production of directives. Modern Language Journal, 90, 473–495.[139] Phakiti, A. (2003). A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading

achievement test performance. Language Testing, 20, 26–56.[140] Pisanski Peterlin, A. (2005). Text-organising metatext in research articles: An English–Slovene contrastive analysis.

English for Specific Purposes, 24, 307–319.[141] Potowsky, K. (2004). Student Spanish use and investment in a dual immersion classroom: Implications for second

language acquisition and heritage language maintenance. Modern Language Journal, 88, 75–101.[142] Precht, K. (1998). A cross-cultural comparison of letters of recommendation. English for Specific Purposes, 17, 241–265.[143] Pritchard, R. M. O., & Nasr, A. (2004). Improving reading performance among Egyptian engineering students:

Principles and practice. English for Specific Purposes, 23, 425–445.[144] Rea–Dickins, P., & Gardner, S. (2000). Snares and silver bullets: Disentangling the construct of formative

assessment. Language Testing, 17, 215–243.[145] Reiter, R. M. R., Rainey, I., & Fulcher, G. (2005). A comparative study of certainty and conventional indirectness:

Evidence from British English and peninsular Spanish. Applied Linguistics, 26, 1–31.[146] Rifkin, B. (1998). Gender representation in foreign language textbooks: A case study of textbooks of Russian.

Modern Language Journal, 82, 217–236.[147] Riley, G. L., & Lee, J. F. (1996). A comparison of recall and summary protocols as measures of second language

reading comprehension. Language Testing, 13, 173–189.[148] Rinnert, C., & Kobayashi, H. (2001). Differing perceptions of EFL writing among readers in Japan.Modern Language

Journal, 85, 189–209.[149] Robinson–Stuart, G., & Nocon, H. (1996). Second culture acquisition: Ethnography in the foreign language

classroom. Modern Language Journal, 80, 431–449.[150] Rogerson–Revell, P. (2008). Participation and performance in international business meetings. English for Specific

Purposes, 27, 338–360.[151] Ross, S. J. (2005). The impact of assessment method on foreign language proficiency growth. Applied Linguistics, 26,

317–342.[152] Rowley–Jolivet, E., & Carter–Thomas, S. (2005). Genre awareness and rhetorical appropriacy: Manipulation of

information structure by NS and NNS scientists in the international conference setting. English for Specific Purposes,24, 41–64.

[153] Rydland, V., &Aukrust, V. G. (2005). Lexical repetition in second language learners’ peer play interaction. LanguageLearning, 55, 229–274.

[154] Saif, S. (2006). Aiming for positive washback: A case study of international teaching assistants. Language Testing, 23,1–34.

[155] Saito,H., &Beecken,M. (1997). An approach to instruction of pragmatic aspects: Implications of pragmatic transferby American learners of Japanese. Modern Language Journal, 81, 363–377.

[156] Sasaki, M. (2000). Effects of cultural schemata on students’ test-taking processes for cloze tests: A multiple datasource approach. Language Testing, 17, 85–114.

[157] Sasaki, M. (2004). Amultiple-data analysis of the 3.5-year development of EFL student writers. Language Learning, 54,525–582.


[158] Sasaki, M. (2007). Effects of study-abroad experiences on EFL writers: A multiple-data analysis. Modern LanguageJournal, 91, 602–620.

[159] Sasaki, M., & Hirose, K. (1996). Explanatory variables for EFL students’ expository writing. Language Learning, 46,137–174.

[160] Sato, K., & Kleinsasser, R. C. (1999). Communicative language teaching (CLT): Practical understandings. ModernLanguage Journal, 83, 494–517.

[161] Scales, J., Wennerstrom, A., Richard, D., & Wu, S. H. (2006). Language learners’ perceptions of accent. TESOLQuarterly, 40, 715–738.

[162] Schauer, G. A. (2006). Pragmatic awareness in ESL and EFL contexts: Contrast and development. LanguageLearning, 56, 269–318.

[163] Schmitt, N., Schmitt, D., Clapham, C. (2001). Developing and exploring the behaviour of two new versions of theVocabulary Levels Test. Language Testing, 18, 55–88.

[164] Shekari, M., & Tahririan, M. H. (2006). Negotiation of meaning and noticing in text-based online chat. ModernLanguage Journal, 90, 557–573.

[165] Shimizu, H., & Green, K. E. (2002). Japanese language educators’ strategies for and attitudes toward teaching kanji.Modern Language Journal, 86, 227–241.

[166] Simpson, R., &Mendis, D. (2003). A corpus-based study of idioms in academic speech.TESOLQuarterly, 37, 419–441.[167] Slimani–Rolls, A. (2005). Rethinking task-based language learning: What we can learn from the learners. Language

Teaching Research, 9, 195–218.[168] Smith, H. J. (2007). The social and private worlds of speech: Speech for inter- and intra-mental activity. Modern

Language Journal, 91, 341–356.[169] So–mui, F. L., & Mead, K. (2000). An analysis of English in the workplace: The communication needs of textile and

clothing merchandisers. English for Specific Purposes, 19, 351–368.[170] Strauss, S., Lee, J., & Ahn, K. (2006). Applying conceptual grammar to advanced-level language teaching: The case of

two completive constructions in Korean. Modern Language Journal, 90, 185–209.[171] Suzuki, M. (2008). Japanese learners’ self revisions and peer revisions of their written compositions in English.

TESOL Quarterly, 42, 209–233.[172] Swain, M., & Lapkin, S. (1998). Interaction and second language learning: Two adolescent French immersion

students working together. Modern Language Journal, 82, 320–337.[173] Swales, J.M., Ahmad,U. K., Chang, Y.–Y., Chavez, D., Dressen, D. F., & Seymour, R. (1998). Consider this: The role of

imperatives in scholarly writing. Applied Linguistics, 19, 97–121.[174] Tomiyaka, M. (2000). Child second language attrition: A longitudinal case study. Applied Linguistics, 21, 304–332.[175] Toth, P. D. (2006). Processing instruction and a role for output in second language acquisition. Language Learning,

56, 319–385.[176] Toth, P. D. (2008). Teacher- and learner-led discourse in task-based grammar instruction: Providing procedural

assistance for L2 morphosyntactic development. Language Learning, 58, 237–283.[177] Towell, R., Hawkins, R., & Bazergui, N. (1996). The development of fluency in advanced learners of French. Applied

Linguistics, 17, 84–119.[178] Tsang, W.–K. (1996). Comparing the effects of reading and writing on writing performance. Applied Linguistics, 17,

210–233.[179] Tudini, V. (2007). Negotiation and intercultural learning in Italian native speaker chat rooms. Modern Language

Journal, 91, 577–601.[180] Turnbull, M. (1999). Multi-dimensional project-based teaching in French Second Language (FSL): A process–

product case study. Modern Language Journal, 83, 548–568.[181] Turner, C. E., & Upshur, J. A. (2002). Rating scales derived from student samples: Effects of the scale maker and the

student sample on scale content and student scores. TESOL Quarterly, 36, 49–70.[182] Uiterwijk, H., & Vallen, T. (2005). Linguistic sources of item bias for second generation immigrants in Dutch tests.

Language Testing, 22, 211–234.[183] VanDaalen–Kapteijns,M.M., Elshout–Mohr,M., &DeGlopper, K. (2001). Deriving themeaning of unknownwords

from multiple contexts. Language Learning, 51, 145–181.[184] Van den Branden, K. (1997). Effects of negotiation on language learners’ output. Language Learning, 47, 589–636.[185] Vandergrift, L. (2003). Orchestrating strategy use: Toward amodel of the skilled second language listener.Language

Learning, 53, 463–496.[186] Vavrus, F. (2002). Postcoloniality and English: Exploring language policy and the politics of development in

Tanzania. TESOL Quarterly, 36, 373–397.[187] Vergaro, C. (2004). Discourse strategies of Italian and English sales promotion letters. English for Specific Purposes, 23,

181–207.[188] Verspoor, M., & Lowie, W. (2003). Making sense of polysemous words. Language Learning, 53, 547–586.[189] Verspoor, M., Lowie, W., & van Dijk, M. (2008). Variability in second language development from a dynamic systems

perspective. Modern Language Journal, 92, 214–231.[190] Vickers, C. J. (2007). Second language socialization through team interaction among electrical and computer

engineering students. Modern Language Journal, 91, 621–640.[191] Walters, F. S. (2007). A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence.

Language Testing, 24, 155–183.


[192] Watanabe, Y. (1996). Does grammar translation come from the entrance examination? Preliminary findings fromclassroom-based research. Language Testing, 13, 318–333.

[193] Watanabe, Y., & Swain, M. (2007). Effects of proficiency differences and patterns of pair interaction on secondlanguage learning: Collaborative dialogue between adult ESL learners. Language Teaching Research, 11, 121–142.

[194] Wei, L. (2000). Unequal election of morphemes in adult second language acquisition. Applied Linguistics, 21, 106–140.

[195] Weir, C. J., &Wu, R.W. (2006). Establishing test form and individual task comparability: A case study of a semi-directspeaking test. Language Testing, 23, 167–197.

[196] Wen, Q., & Johnson, R. K. (1997). L2 learner variables and English achievement: A study of tertiary-level Englishmajors in china. Applied Linguistics, 18, 27–48.

[197] Wilhelm, K. H. (1997). Sometimes kicking and screaming: Language teachers-in-training react to a collaborativelearning model. Modern Language Journal, 81, 527–542.

[198] Xu, Y., Gelfer, J., & Perkins, P. (2005). Using peer tutoring to increase social interactions in early schooling. TESOLQuarterly, 39, 83–106.

[199] Yamashita, J. (2003). Processes of taking a gap-filling test: Comparison of skilled and less skilled EFL readers.Language Testing, 20, 267–293.

[200] Yu, G. (2007). Students’ voices in the evaluation of their written summaries: Empowerment and democracy for testtakers? Language Testing, 24, 539–572.

[201] Yu, G. (2008). Reading to summarize in English and Chinese: A tale of two languages? Language Testing, 25, 521–551.[202] Yu, M.–C. (2004). Interlinguistic variation and similarity in second language speech act behavior.Modern Language

Journal, 88, 102–119.[203] Yu, M.–C. (2008). Teaching and learning sociolinguistic skills in university EFL classes in Taiwan. TESOL Quarterly,

42,31–53.[204] Zhang, Y. (2004). Processing constraints, categorical analysis, and the second language acquisition of the Chinese

adjective suffix –de(ADJ). Language Learning, 54, 437–468.[205] Zhao, Y. (1997). The effects of listeners’ control of speech rate on second language comprehension. Applied

Linguistics, 18, 49–68.


Documents

Mixed methods research: Toward new research designs in applied linguistics