The Value of Optimising Quantitative Instrument

Development via Qualitative Techniques in

Political Science Research

Stefania Kalogeraki1

Paper for the 6th ECPR General Conference

Reykjavík, Iceland, 25 - 27 August 2011

This paper is work in progress,

please do not cite it. Comments are more than welcome.

1 Stefania Kalogeraki is an elected lecturer in Quantitative Methods in Sociology in the Department ofSociology of the University of Crete (Greece). She has BSc in Statistics (Athens University inEconomic and Business, Greece), MA and PhD in Sociology (Reading University, UK). She hasparticipated in European and Greek research projects and published articles in international and Greekjournals. Her main research interests include quantitative methods in social research with a specialfocus on measurement issues and social demography.E-mail: kaloge@social.soc.uoc.gr

2

AbstractThe optimization of the development of quantitative instruments is a key concern inany social science endeavour. Whilst instrument development has been for manyyears restricted to mono-method approaches, a new era has come that combinesqualitative techniques to enhance the development of quantitative instruments. Thisapproach constitutes one of the main rationales of conducting mixed method studieswhich in political science research, in contrast to the adjacent social sciencedisciplines, has been scarcely applied. The paper’s rationale is two-fold; a) to reviewthe three-phase exploratory design for the optimization of the development ofquantitative instruments via qualitative techniques and b) to present the main benefitsand challenges of the approach for political science research. Instrument developmentvia qualitative techniques applies an inductive-deductive and an emic-etic perspectivethat increases construct-related validity in cross-sectional political science studies andeliminates construct bias in cross-national and cross-cultural ones. The paper’s mainthesis is that the mixed method approach is not the panacea of all research inquiries.However, the optimization of quantitative instrument development via qualitativetechniques may adequately serve mixed method’s fundamental principal, i.e.,maximizing the potency and minimizing the weaknesses derived from theamalgamation of the two methods to enhance the validity of political studies’conclusions.

Introduction

Mixed method studies have become increasingly popular in social sciences providing

evidence that scholars utilize multiple perspectives in order to examine phenomena in

a more eclectic manner (Tashakkori & Creswell, 2008; Tashakkori & Teddlie, 2010).

The mixed method approach bridges across the schism between the qualitative and

quantitative research traditions (Teddlie & Tashakkori, 2003) by offering a variety of

choices, options and approaches (Johnson & Onwuegbuzie, 2004). Johnson,

Onwuegbuzie and Turner (2007) based on the analysis of 19 mixed methods’

definitions given from academic leaders provide the following definition:

Mixed methods research is the type of research in which a researcher or teamof researchers combines elements of qualitative and quantitative researchapproaches (e.g., use of qualitative and quantitative viewpoints, datacollection, analysis, inference techniques) for the broad purposes of breadthand depth of understanding and corroboration. (Johnson, Onwuegbuzie &Turner, 2007, p. 123)

The fundamental principle of mixed method research is based on the collection of

multiple data using different approaches and methods in such a way that the resulting

amalgamation is likely to yield a research outcome of complementary strengths and

3

non-overlapping weaknesses (Johnson & Turner, 2003). Under this framework, mixed

method research is likely to produce a research outcome that is superior to mono-

method studies (Johnson & Onwuegbuzie, 2004). Philosophically, it is the “third

methodological movement” (Tashakkori, & Teddlie, 2003, p. x) that involves the

pragmatic method and system of philosophy. It combines three logics of inquiry by

linking the deductive logic of traditional quantitative research (confirmation,

theory/hypothesis testing), the inductive logic of qualitative research (discovery,

exploration, theory/hypothesis generation) and the abductive one (generative, creative

problem solving) (Johnson & Onwuegbuzie, 2004). Pragmatism has emerged as a

common alternative to the either/or choice of positivism and constructivism of the

quantitative and qualitative approach, respectively offering an epistemological

approach that justifies the integration of methods in order for researchers to best

frame, address and provide logical and practical answers to their research questions.

Mixed method research is a relatively new approach, compared to the qualitative and

quantitative traditions, with a short history of approximately 20 years (Greene, 2008).

Its history is even shorter in the field of disciplines such as political science and

comparative politics that so far have not participated much in mixed method research

discussions (Harrits, 2011).

Tarrow’s (1995) work constitutes one of the first steps of bridging quantitative and

qualitative methods in the field of political science by providing some guidance on

how to blend the two traditions. For instance, he highlights the importance of

qualitative approach for process tracing1, for exploring the ‘tipping points’ that play a

critical role in shaping long-term processes of change and for providing deeper

insights into findings derived from quantitative studies. In turn, he emphasizes the

importance of quantitative methods to best frame and to generalize qualitative studies’

findings. In Tarrow’s (1995) view the most valuable interaction between qualitative

and quantitative methods takes place when scholars triangulate among alternative

methods and data in order to answer the same research question. Whilst Tarrow

(1995) provides a general framework of conducting mixed method studies, the

author’s overview is not exhaustive excluding among others the crucial role mixed

methods play in instrument development processes.

1 Process tracing refers to the application of qualitative techniques to investigate procedures of changewithin cases providing detailed evidence on causal mechanisms and causal processes that underliequantitative findings.

4

A few years later, Bennett and Braumoeller (2002) emphasized the role of combining

quantitative and qualitative techniques in political science research for theory testing

and theory development. In the former, the mixed method approach is applied to

determine whether the qualitative findings can be generalized to the wider population

or whether they can be used to uncover the causal mechanisms behind correlations. In

theory generating mixed method research, the outliers derived from quantitative

studies can be qualitatively analyzed to provide information for identifying omitted

variables. Bennett and Braumoeller’s (2002) arguments in theory generation mixed

method studies, imply that the specific approach may reveal potential measurement-

related problems in political science research by uncovering left-out variables through

qualitative analysis of outlier quantitative data.

The significance of the mixed method approach in enhancing the quality of

measurements in comparative political science research has been outlined in

Lieberman’s (2005) nested design which involves the specification of different paths

in the combination of statistical (large N analysis-LNA) and case study (small N

analysis-SNA) techniques. Lieberman (2005, p.436) argues that the main scope of the

specific design is to “improve the quality of conceptualization and measurement,

analysis of rival explanations, and overall confidence in the central findings of a

study”. The nested design allows the evaluation and/or development of a survey

instrument for the measurement of a large number of cases (LNA) through close

range analysis of one or a few cases (SNA) with the rationale of increasing the

explanatory power and the robustness of the former. Whilst Lieberman (2005)

advocates that the amalgamation of the advantages of SNA and LNA associated with

internal and external validity, respectively result in the enhancement of the quality of

the concepts and measurements, the nested design has been criticised by different

scholars.

For instance, Mastenbroek and Doorenspleet (2007) argue that Lieberman’s (2005)

nested strategy does not always result in the improvement of measurements unless it

is applied in a comparison of most similar systems, i.e., of cases with great similarities

(such as similar countries). In addition, Harrits (2010) develops some criticism against

the nested design by advocating that Lieberman’s (2005) thesis with respect to the

conceptualization of SNA, as a qualitative approach that focuses on the analysis of

one or a few intrinsically interesting entities, is inadequate for individual-level

5

behaviours. Hence, SNA of close range analysis is likely to be inappropriate for

evaluating and/or developing instruments measuring attitudes in large scale surveys.

Whilst optimizing instrument development and confronting measurement-related

issues via mixed method approaches has been an issue of concern in political science

research, the utilization of qualitative techniques to enhance the development of

quantitative instruments has not been fully investigated. However, the specific issue

has been adequately explored in the adjacent fields of social science research

providing typologies that underline that among the main rationales of conducting

mixed method studies is the optimization of instrument development. For instance,

Greene, Caracelli and Grahamet (1989) in their five-purpose typology include mixed

method studies with the rationale to use sequentially the results from one method to

help develop and inform the other. More specifically, they argue that “One method is

implemented first, and the results are used to help select the sample, develop the

instrument, or inform the analysis for the other method” (Greene, Caracelli &

Grahamet, 1989, p.267). Furthermore, Collins, Onwuegbuzie and Sutton (2006)

develop a four-fold typology of mixed method rationales involving among others

instrument fidelity. Instrument fidelity refers to the assessment of the appropriateness

and/or utility of existing instruments and the creation and improvement of new ones.

The development of quantitative instruments and their validation through multiple

research methods has a long history dated back to Campbell and Fiske’s (1959)

framework of multiple traits and multiple methods (MTMM). The approach involves

multiple operationalism by using at least three traits measured with three different

methods, leading to nine different observed variables for validation purposes. Whilst

some scholars advocate that Campbell and Fiske’s (1959) MTMM framework provide

the impetus for the concept of triangulation for validating purposes (Johnson,

Onwuegbuzie & Turner, 2007) others suggest that MTMM’s underling principal is

that quantitative methods are sufficient for the development of quantitative

instruments (Onwuegbuzie, Bustamante & Nelson, 2010).

Despite the debates on MTMM’s main rationale, the amalgamation of qualitative and

quantitative methods in instrument development is likely to produce a research

outcome that is superior to quantitative mono-method approaches (Onwuegbuzie,

Bustamante & Nelson, 2010), as the former optimize the conceptualization process

and enhance measurements’ internal validity. Such a mixed method approach

involves an inductive-deductive-abductive logic and an emic-etic perspective that

6

increases measurement-related validity in cross-sectional and cross-national/cross-

cultural research in social science in general and specifically in political science.

Under this framework, the main rationale of the paper is two-fold; to review one of

the mixed method designs applied to the optimization of the development of

quantitative instruments via qualitative techniques named sequential exploratory

three-phase design and to present its main benefits and challenges for political science

research.

Sequential exploratory three-phase design

Creswell (1999, p.460) highlights the fundamental role mixed methods play in

quantitative instrument development via qualitative techniques by advocating that

among the most crucial purposes of the mixed method approach is to develop

“Quantitative measures and instruments grounded in the views of subjects or

participants”. Such a mixed method approach is extremely valuable in cases where

there are no available measures, the existing ones are poorly conceptualized and when

the available measures do not represent the population being studied (Creswell, 1999;

Creswell, Fetters & Ivankova, 2004; Creswell & Plano Clark, 2011).

Creswell (1999) is among the first scholars to introduce a sequential exploratory

design with the rationale of instrument development. The design has been recently

enriched including three phases and a detailed framework of how to apply qualitative

techniques in order to construct rigorous quantitative instruments (Creswell & Plano

Clark, 2011). The initial phase of the design is a qualitative one as the most adequate

to explore a phenomenon (Creswell et al., 2003). The second phase represents the

point of interface in mixing, as mixed method researchers build on the result of the

qualitative phase to develop the quantitative instrument. The final phase includes the

testing and administration of the quantitative instrument (Creswell & Plano Clark,

2011).

Although the three-phase exploratory design seems quite straightforward it has its

own challenges. Researchers applying such a mixed method approach need to decide

the sampling procedures applied in each phase, the most appropriate data to use from

the qualitative phase to build the quantitative instrument and the processes of

designing an instrument with good psychometric properties. The sampling procedures

in the qualitative and quantitative approach of the sequential exploratory design are

different as the participants in the former are not the same as the ones in the

7

quantitative follow-up study which involves large sample sizes with the rationale of

inferences to the population studied (Creswell & Plano Clark, 2011).

With respect to the challenge of determining the qualitative data to be used for

instrument generation, mixed method scholars need to carefully identify and assess

useful quotes or sentences, coding segments of information and the grouping of codes

into broad themes (Morse & Niehaus, 2009). In this process, the broad themes derived

from the qualitative analysis can be applied for constructing the scales to be

measured; the individual codes within each theme can be used for the variables and

the individuals’ quotes for specific items or questions (Creswell & Plano Clark,

2011). Oppenheim (1992) suggests that at the initial stage of item development the

over inclusion of items derived from the qualitative analysis is essential to ensure that

all the aspects or dimensions of the topic are covered.

The most fundamental challenge in instrument building process is to generate an

instrument with psychometric stability and feasibility (Vogt, King & King, 2004).

Mixed method researchers need to employ rigorous scale development procedures

addressing validity and reliability considerations (DeVellis, 2003). The former

involve results that accurately reflect the concept being measured, whilst the latter

refer to the consistency of results derived from the same measure (Babbie, 2002).

In the process of scale development, the qualitative findings are used to generate an

item pool and the appropriate scales of measurement are determined which are further

reviewed by an expert panel (DeVellis, 2003). Expert consultation on the

specification of constructs and the review of scaling procedures is considered by

Messick (1995a) to be the sin qua non of measurements’ validity. At this stage, it

might be also useful to include validated items from other scales or instruments

(DeVellis, 2003). Following experts’ consultation, the instrument is administered to a

sample of individuals for further validation that allow the evaluation of the items with

quantitative techniques e.g., item-scale correlations, item variance, reliability (De

Vaus, 1990; Oppenheim, 1992; DeVellis, 2003). The results of the pre-test allow for

the optimization of the instrument and its scales’ length based on the item

performance and reliability checks (DeVellis, 2003). At the final stage of the

sequential exploratory design the instrument is administered to a large sample in order

for researchers to conduct statistical tests and potentially make claims about the

population in question.

8

Benefits of mixed method approach for instrument development

Munk and Verkuilen (2002, p.31) advocate that “the careful development of measures

constitutes the foundation for efforts at drawing causal inferences and is a critical task

in itself”. In the development of measures two processes are crucial; the

conceptualization and the operationalization. The former involves the specification of

observations and measurements that give concepts definite meaning for the rationale

of the study. In addition, conceptualization includes specifying concept’s indicators

and describing its dimensions. Operationalization is considered to be an extension of

conceptualization that denotes the exact procedures that will be used to measure the

variable’s attributes (Babbie, 2002). The translation of theoretical concepts into

related variables as part of the operationalization process is crucial as it bridges the

theoretical and conceptual level to the measurement level (Bergman, 2010). Any

mismatch between the conceptual and the operational definitions constitute a serious

threat to the validity of the measures and consequently to the study’s findings (Vogt,

King & King, 2004).

The assessment of measurement validity in an instrument development process

involves three types of validation; content validity, criterion-related validity

(including predictive and concurrent validity) and construct validity (Cronbach &

Meehl, 1955). Content validity assesses the degree to which the items are relevant to

and representative of the content being measured. According to Cronbach and Meehl

(1955), criterion-related validity assesses the extent to which scores produced by a

measure are correlated with scores of independent/external variables (called criterion

variables) which are considered to measure directly the phenomenon of concern

(concurrent validity) and are useful in predicting future scores representing

hypothetically related phenomena (predictive validity). Construct validity assesses the

extent to which an instrument adequately measures the constructs they are designed to

measure.

The triumvirate of content, criterion and construct validity does not represent three

distinct types of measurement validity but rather reflects a unitary concept that places

construct validity at the centre encompassing all types of measurement-related

validity evidence (Cronbach & Meehl, 1955; Messick, 1989a; 1989b; 1995b). Valid

measurements are derived when scores accurately reflect the intended concept; hence

a prerequisite of valid measurements is the refinement and clarification of the

concepts of concern (Adcock & Collier, 2001).

9

In social science research in general and specifically in political science investigators

are usually concerned with concepts that are more complex to handle. Northrop

(1947) distinguishes between two types of concepts; the concepts-by-intuition and the

concepts-by-postulation. The former involve simple concepts whose meaning is

immediately apparent and involve feelings, judgments, norms and behaviours which

are relatively clear what they mean. On the contrary, concepts-by-postulation or

constructs, as they are also called, involve less obvious concepts and hence require

explicit definitions to be properly understood (Saris & Gallhofer, 2007). Concepts-by-

postulation can be captured by using a set of items that represent simpler concepts,

i.e., concepts-by-intuition. Concepts-by-postulation have been also characterised as

thick concepts due to their multidimensional nature and their measurement through

multiple indicators. Thick concepts (such as “social trust”, “political trust”, “political

efficacy” etc) are rather prevalent in political science research (Coppedge, 1999).

Political democracy is a traditional example of a thick concept that has been

frequently applied in political science research. However, various scholars address the

challenges of political democracy’s measurement in terms of its conceptualization,

operationalization and aggregation specifically when the concept is applied across

diverse contexts (Bollen, 1980, 1990; Coppedge & Reinicke, 1990; Przeworski et al.,

2000; Munk & Verkuilen, 2002;Bowman, Lehoucq & Mahoney, 2005).

The qualitative techniques are more adequate than quantitative ones to measure thick

concepts as they allow for more careful conceptualization enhancing the internal

validity of measurements (Coppedge, 1999). Qualitative scholars are highly

concerned with conceptual validity hence their primary focus is on the development

of precise and clear conceptual definitions (Mahoney & Goertz, 2006; Bergman,

2010). On the contrary, scholars of the quantitative strand sacrifice definitional

complexity by applying thin but broad concepts, i.e., simplistic descriptions of cases

that allow testing generalizations in the larger population (Coppedge, 1999).

Furthermore, in the quantitative tradition the measurement detaches information from

its original ecological “real world” (Moghaddam, Walker, & Harre, 2003), a

phenomenon that has been characterised as decontextualization (Viruel-Fuentes,

2007). On the contrary, the qualitative approach investigates individuals holistically

within their natural environment allowing a fully contextualised approach of rich

accounts of human experiences examined within the original context in which

observations occur (Gelo et al., 2008).

10

The enhancement of quantitative measurements’ internal validity through the initial

qualitative phase of the sequential exploratory design is accomplished as the process

of defining the elements that are relevant to and representative of the construct under

study is grounded in real life situations and observations (Padgett, 1998; Rowan &

Wulff, 2007; Nassar-McMillan et al., 2010). The initial qualitative phase explores

respondents’ variations in the construction of concepts’ meaning informing the

processes of conceptualization and operationalization and yields items characterised

by high degrees of content validity (Haynes, Richard & Kubany, 1995; Vogt, King &

King, 2004; Bergman, 2010).

In addition, the combination of the qualitative and quantitative approaches in a

sequential exploratory design with the rationale of instrument development allows the

incorporation of insiders’ views (stemming from the participants involved in the

instrument development) and the outsiders’ views (stemming from extant theories,

researchers’ a priori assumptions and experts’ consultation) on the specification of the

construct. The insiders’ views involve the emic viewpoint of the group members

whilst the outsiders’ views capture the etic viewpoint (Pike, 1967; Currall & Towler,

2003). By mixing the two viewpoints with the rationale of instrument development an

inside-outside legitimation is accomplished enhancing measurement-related validity

(Onwuegbuzie & Johnson, 2006). Such a mixed method approach not only increases

the validation of the instrument but also reveals additional facets of the phenomenon

under study optimizing research outcomes. As Bergman (2010) argues the systematic

concept analyses through qualitative approaches:

…not only help in validating research instruments and scales, but may gofurther in that they could produce complementary subsets of results, whichwould enrich overall findings (Bergman, 2010, p.172)

Furthermore, instrument development via mixed method studies is particularly useful

when seeking to establish equivalence in the theoretical concepts and their operational

indicators as they are applied for comparisons across different cultural contexts

(Onwuegbuzie, Bustamante & Nelson, 2010) and over time. Equivalence implies that

concepts measure the same latent traits in a way that respondents’ scores on certain

items and scales can be compared in a direct and straightforward manner across

different contexts (Hui & Triandis, 1985; Dogan & Pelassy, 1990; Sartori 1994; Van

de Vijver & Leung , 1997; Van de Vijver, 1998).

11

The validation of an instrument in a specific context does not imply that the

theoretical concepts and their operational indicators can be applied in multiple

environments (MacIntyre, 1971; Messick, 1989b). Diverse understandings of a

concept are likely to result in different operationalizations (Adcock & Collier, 2001).

For instance, the concept of political participation is likely to mean very different

things across different contexts, such as voting in one country or mobilizing activists

against nuclear power in another. Moncagatta’s (2009) paradoxical findings with

respect to the high support of the democratic regime in Venezuela both by

respondents of great and no trust to president Chávez reflect in-equivalence problems

of the concept of “support to democracy” when applied in diverse cultural settings.

Scholars conducting comparisons across different contexts have to ensure that their

instrument measures the same underlying factors otherwise there is a serious threat of

construct in-equivalence and consequently of construct bias in the study’s findings

(Van de Vijver & Leung, 1997). Construct bias takes place when there is only partial

overlap in the definitions of the construct, there is incomplete coverage of the

construct’s facets and when constructs are associated with different behaviours or

characteristics across individuals in different nations, cultures and cultural groups

(Van de Vijver & Tanzer, 2004).

Scholars from comparative political research are highly concerned with the

application of constructs that are vulnerable to in-equivalence (Bunce, 1995; Munk &

Verkuilen, 2002). For instance, Bollen (1990) emphasises the need to optimize the

measurement of political democracy in comparative political research by further

clarifying its meaning and applying new measures. Similarly, Munk and Verkuilen

(2002) offer a comprehensive framework with the rationale of improving democracy’s

conceptualization and measurement in cross-national studies.

Instruments that are applied for comparisons over time are also susceptible to in-

equivalence, as the meaning of different constructs and the items used to measure

them might change as time passes. For instance, Baumgartner and Walker (1990)

emphasise the need of a new measurement of group membership in the United States

as the measures applied reflect conceptualizations that are appropriate for previous

generations. Furthermore, Elkins and Sides (2009) advocates that comparisons of

democracy over time potentially exhibit serious issues of non-equivalence.

Whilst there are statistical tools to diagnose in-equivalence problems and

consequently construct bias, these techniques are inadequate to trace and tackle the

12

sources of such bias (Elkins & Sides, 2009). Van de Vijver (2011) highlights the

pivotal role the mixed method approach plays on tracing and tackling the sources of

in-equivalence in specific items through the careful re-conceptualization and the

deeper knowledge of the context that a concept is investigated, i.e., processes that are

adopted in the initial qualitative phase of the sequential exploratory design. The same

author advocates that the qualitative approach in the development of an instrument

that is applied across different contexts helps researchers to evaluate the

appropriateness of the concept definition, identify missing concepts, employ

appropriate question wording and create suitable response formats (Van de Vijver,

2011).

Challenges of mixed method approach for instrument development

In the previous section it was advocated that instrument development via the

sequential exploratory design increases measurement’s internal validity as the initial

qualitative phase allows for more careful concept construction specifically of thick

multidimensional and complex concepts frequently applied in political science

research (Coppedge, 1999). However, such an increase in internal validity is done at

the cost of external validity of the measurement as the qualitative phase involves

results derived from a small number of cases that do not represent universally

applicable, comparable and generalizable concepts. Qualitative scholars in defense of

such criticism would advocate that their quantitative counterparts have to deal with

the opposite challenge. The increase of external validity through the application of

broad but thin concepts for testing generalizations in the larger population is done at

the cost of internal validity as the simplistic indicators applied in the quantitative

analysis do not capture all the relevant aspects of thick multidimensional concepts

(Coppedge, 1999; Munck & Verkuilen, 2002; Bowman, Lehoucq & Mahoney, 2005).

The diverse focus on internal and external validity of the qualitative and quantitative

strand respectively reflects the chasm that exists in the two traditions with respect to

measurement issues in social science research in general, and specifically in political

science (Mahoney & Goertz, 2006). On the one hand, qualitative scholars attempt to

minimize measurement error by concentrating on conceptual validity; on the other

hand quantitative scholars consider that measurement error occurs at the level of

indicators and not at the conceptual one. Hence, the latter focus on modeling

13

measurement errors and modifying indicators rather than developing processes

associated with concept revision.

The most crucial challenge in the application of qualitative techniques for optimizing

instrument development is to bridge the two traditions with the rationale of enhancing

simultaneously the internal and external validity of measurements. The three-phase

sequential exploratory design is likely to produce such an outcome as the initial

qualitative phase increases the internal validity of the measurement by refining the

concept of concern (Haynes, Richard & Kubany, 1995; Padgett, 1998; Vogt, King &

King, 2004; Rowan & Wulff, 2007; Bergman, 2010); whilst the second phase through

the expert panel consultation and the quantitative analysis of the pre-test results

optimizes the external validity of the measurement based on the item performance and

reliability checks (De Vaus, 1990; Oppenheim, 1992; Messick, 1995a; DeVellis,

2003).

The sequential exploratory design has been applied in numerous studies exploring

diverse social phenomena, verifying that the challenges of mixing the two traditions

with the rationale of instrument optimization can be adequately confronted in

empirical research (Meijer, Verloop & Beijaard, 2001; Mak & Marshall, 2004;

Durham, Tan & White, 2011). For instance, the specific design has been applied for

developing instruments that measure perspectives of academia held by school

psychology graduate students (Nagle et al., 2004), the nature of domestic violence

against women (Smith, Earp & DeVellis, 1995), the shopping motivations (Arnold &

Reynolds, 2003), the organizational assimilation in diverse industries (Myers &

Oetzel; 2003), the factors affecting changes in the size of graduate programs (Milton

et al., 2003), and the health promoting lifestyle behaviours among Japanese college

women (Tashiro, 2002).

Mixed method scholars in the process of instrument development for comparisons

across different contexts are likely to face additional challenges associated with the

conceptual stretching which arises when the concept of concern is applied to cases for

which, by relevant scholarly standards, it is inappropriate (Sartori, 1970; Collier &

Mahon, 1993). Sartori (1970) argues that concepts are formed at different levels of

abstraction (or generality) due to the trade off between concept’s “intension” (i.e., the

number of defining attributes) and “extension” (i.e., the number of cases it can be

applied). Qualitative scholars are more concerned with the more detailed contextual

analyses working with a high level of concept’s intension (i.e., incorporate relatively

14

complex conceptual definitions), at the cost though of its extension (i.e., involve a

limited number of cases). Meanwhile, quantitative scholars develop concepts of high

extension (i.e., involve relatively big sample sizes) applying comparable and

generalizable concepts; at the cost though of the concepts’ intension (i.e., apply

relatively simplistic indicators).

When concepts travel across different contexts, then scholars in order to avoid

conceptual stretching have to move up the ladder of abstraction (or generality), i.e., to

increase the extension of the concept by redefining it as something broader and more

applicable to different contexts (Sartori, 1970). Such a climbing, on the one hand,

ensures the concept’s comparability over time and space but on the other the concept

per se is likely to become very imprecise and abstract. In order to confront with

conceptual stretching, Sartori (1970) suggests that scholars may opt to make ‘bounded

generalizations’ for comparisons between relatively homogenous cases. Alternative

solutions to cope with conceptual stretching is to develop appropriate subtypes of the

concept of concern; for instance, researchers can develop appropriate subtypes of

democracy that simultaneously allow them to capture diverse forms of democracy and

avoid conceptual stretching (Collier & Levitsky, 1997).

An additional solution to conceptual stretching, although not so welcome by Sartori

and his followers (Lindberg, 2009), is to redefine the concept of interest by

eliminating some of its characteristics and adding new ones. Whilst it is feasible to

accomplish this alternative with the application of the sequential exploratory design,

recently more advanced mixed method designs have been developed for instruments

applied across diverse cultural settings. For instance, Onwuegbuzie, Bustamante and

Nelson (2010) present a ten-step mixed method process for instrument development

and construct validation through multiple data sources and crossover analyses for

cross-cultural comparisons.

Discussion

The optimization of instrument development is a key concern in any social science

endeavour. Quantitative tools that are valid in their indented purpose and measure the

concepts they are intended to measure possess the requisite psychometric properties to

provide valid conclusions and to lead to advances in theory and practice (Cronbach &

Meehl, 1955; Messick, 1989b; Messick, 1995a; DeVellis, 2003; Groves et al., 2009;

Bergman, 2010). Whilst there is a massive literature on quantitative measures,

15

indicators and scales, quantitative scholars insist on focusing whether the items

represent the construct of concern as measured by reliability and validity scores

disregarding the origins of questionnaire items (Rowan & Wulff, 2007; Nassar-

McMillan et al., 2010). The crucial process of item generation becomes the major

concern of qualitative scholars who focus on conceptual validity in contrast to their

quantitative counterparts who concentrate on measurement validity by modeling

measurement errors and modifying indicators (Mahoney & Goertz, 2006) as the

mathematical statistics applied are not designed to tackle measurement errors on the

conceptual level (Jacoby, 1991). However, the process of conceptualization is crucial

in instrument development, as Lazarsfeld and Barton (1965) advocated a few decades

ago

Before we can investigate the presence or absence of some attribute ... orbefore we can rank objects or measure them in terms of some variable, wemust form the concept of that variable. (Lazarsfeld & Barton, 1965, p.155)

Whilst the chasm between qualitative scholars’ concern for substantively valid

concepts and the quantitative scholars’ interest in good numerical measures seems

unbridgeable, the sequential three-phase exploratory design is likely to yield

instruments with high levels of internal and external validity. Such a design allows

mixed method scholars to shift from the constructivist principles of the initial

qualitative phase that involve multiple perspectives and deeper understanding of the

concept of concern to the positivist principles of quantitative analysis of measuring

variables and statistical trends (Creswell & Plano Clark, 2011). Hence, instead of

relying on deductive reasoning and general premises to reach specific conclusions or

inductive approaches that seek general conclusions based on specific premises an

abductive logic is adopted that simultaneously allows the generation of a practical

research solution and the integration of various theories and approaches (Tomiyamal

et al., 2003) with the rationale of developing valid and reliable measurements. The

sequential exploratory design employs a pragmatic approach that sidesteps the debates

between the existence of objective ‘truth’ and the value of subjective perceptions and

balances the etic viewpoint of outsiders’ views (i.e., researchers, expert panels) with

the emic one derived from real life observations. Such a mixed method approach is

likely to produce a rigorous instrument that is superior to using exclusively

quantitative techniques for instrument development (Onwuegbuzie, Bustamante &

Nelson, 2010).

16

Whilst mixed method approaches is not the panacea of all research inquiries the

sequential exploratory design with the rationale of instrument development

adequately serves mixed method’s fundamental principal i.e., maximizing the potency

and minimizing the weaknesses derived from the amalgamation of qualitative and

quantitative techniques (Johnson & Turner, 2003) to enhance validity in political

science research conclusions.

References

Adcock, R., & Collier, D. (2001). Measurement validity: A shared standard forqualitative and quantitative research. American Political Science Review, 95(3), 529-546.

Arnold, M., & Reynolds, K. (2003). Hedonic shopping motivations. Journal ofRetailing, 79(2), 77-95.

Babbie, E. (2002). The Basics of Social Research (2nd ed.). Belmont, CA:Wadsworth/Thomson Learning.

Baumgartner, F., & Walker, J. (1990). Response to Smith’s ‘Trends in voluntarygroup membership: Comments on Baumgartner and Walker’: Measurement validityand the continuity of research in survey research. American Journal of PoliticalScience, 34, 662-670.

Bennett, A., & Braumoeller, B. (2002). Where the model frequently meets the road:Combining statistical, formal, and case study methods. Paper presented at the AnnualMeeting of the American Political Science Association, 28 August, Boston,Massachusetts.

Bergman, M. (2010). On concepts and paradigms in mixed methods research.Journal of Mixed Methods Research, 4(3), 171-175.

Bollen, K. (1980). Issues in the comparative measurement of political democracy.American Sociological Review, 45(3), 370-390.

Bollen, K. (1990). Political democracy: Conceptual and measurement traps. Studies inComparative International Development, 25(1),7-24.

Bowman, K., Lehoucq, F., & Mahoney, J. (2005). Measuring political democracy:Case expertise, data adequacy, and Central America. Comparative Political Studies,38, 939-970.

Bunce, V. (1995). Should transitologists be grounded? Slavic Review, 54(1), 111-127.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by themultitrait multimethod matrix. Psychological Bulletin, 56, 81-105.

17

Collier, D., & Levitsky, S. (1997). Democracy with adjectives: Conceptual innovationin comparative research. World Politics, 49, 430-451.

Collier, D., & Mahon, J. (1993). Conceptual stretching revisited: Adapting categoriesin comparative analysis. American Political Science Review, 87, 845-855.

Collins, K., Onwuegbuzie, A., & Sutton, I. (2006). A model incorporating therationale and purpose for conducting mixed methods research in special education andbeyond. Learning Disabilities: A Contemporary Journal, 4, 67-100.

Coppedge, M. (1999). Thickening thin concepts and theories: Combining large-N andsmall in comparative politics. Comparative Politics, 31, 465-476.

Coppedge, M., & Reinicke, W. (1990). Measuring polyarchy. Studies in ComparativeInternational Development, 25(1), 51-72.

Creswell, J. (1999). Mixed-Method research: Introduction and application. In G.Cizek (Ed.), Handbook of Educational Policy, (pp. 455-472). San Diego: AcademicPress.

Creswell, J., Fetters M., & Ivankova N. (2004). Designing a mixed methods study inprimary care. Annals of Family Medicine, 2(1), 7-12.

Creswell, J., & Plano Clark, V. (2011). Designing and Conducting Mixed MethodsResearch (2nd ed.). Thousand Oaks, CA: Sage.

Creswell, J., Plano Clark, V., Guttman, M., & Hanson, W. (2003). Advanced mixedmethods research designs. In A. Tashakkori, & C. Teddlie, (Eds.), Handbook ofMixed Methods in Social and Behavioral Research, (pp.209-240). Thousand Oaks,CA: Sage.

Cronbach, L., & Meehl, P. (1955). Construct validity in psychological tests.Psychological Bulletin, 52(4), 281-302.

Currall, S., & Towler, A. (2003). Research methods in management andorganizational research: Toward integration of qualitative and quantitative techniques.In A. Tashakkori, & C. Teddlie (Eds.), Handbook of Mixed Methods in Social andBehavioral Research, (pp. 513-526). Thousand Oaks, CA: Sage.

De Vaus, D. (1990). Surveys in Social Research (2nd ed.). Sydney: Allen & Unwin.

DeVellis, R. (2003). Scale Development (2nd ed.). Thousand Oaks, CA: Sage.

Dogan, M., & Pelassy, D. (1990). How to Compare Nations: Strategies inComparative Politics (2nd ed.). Chatham, NJ: Chatham House.

Durham, J., Tan B.-K., & White, R. (2011).Utilizing mixed research methods todevelop a quantitative assessment tool: An example from explosive remnants of a warclearance program. Journal of Mixed Methods Research, 1-15.

18

Elkins, Z., & Sides J. (2009). The vodka is potent but the meat is rotten: Evaluatingmeasurement equivalence across contexts. Paper presented at the October 30 JuniorFaculty Research Colloquium, Department of Political Science, George WashingtonUniversity, Washington, DC. [Online] Available at:http://home.gwu.edu/~jsides/vodka.pdf (15 July 2011)

Gelo, O., Braakman, D., & Benetka, G. (2008). Quantitative and qualitative research:Beyond the debate. Integrative Psychological & Behavioral Science, 42, 266-290.

Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology?Journal of Mixed Methods Research, 2, 7-22.

Greene, J., Caracelli, V., & Graham, W. (1989). Toward a conceptual framework formixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11,255-274.

Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R.(2009). Survey Methodology (2nd edn). New York: John Wiley & Sons.

Harrits, G. (2010). Understanding and explaining political action. A mixed methodstrategy. Political Methodology - Concepts and Methods Working Papers Series, 17.Available at:http://www.concepts-methods.org/Files/WorkingPaper/PM%2027%20Harrits.pdf (29June 2011)

Harrits, G. (2011). More than method? A discussion of paradigm differences withinmixed methods research. Journal of Mixed Methods Research, 5(2) 150-166.

Haynes, S. Richard, D., & Kubany, E. (1995). Content validity in psychologicalassessment: A functional approach to concepts and methods. PsychologicalAssessment, 7, 238-247.

Hui, C., & Triandis, H. (1985). Measurement in cross-cultural psychology. A reviewand comparison of strategies. Journal of Cross-cultural Psychology, 16, 131-152.

Jacoby, W. (1991). Data Theory and Dimensional Analysis. Newbury Park, CA:Sage.

Johnson, R., & Onwuegbuzie, A. (2004). Mixed methods research: A researchparadigm whose time has come. Educational Researcher, 33(7), 14-26.

Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixedmethods research. Journal of Mixed Methods Research, 1(2), 112-133.

Johnson, R., & Turner, L. (2003). Data collection strategies in mixed methodsresearch. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods inSocial and Behavioral Research, (pp. 297-319). Thousand Oaks, CA: Sage

19

Lazarsfeld, P., & Barton, A. (1965). Qualitative measurement in the social sciences:Classification, typologies, and indices. In D. Lerner & H. Lasswell (Eds.), The PolicySciences (pp.155-192). Stanford, California: Stanford University Press.

Lieberman, E. (2005). Nested analysis as a mixed-method strategy for comparativeresearch. American Political Science Review, 99, 435-452.

Lindberg, S. (2009). Accountability :The core concept and its subtypes. Africa Powerand Politics Working Paper,1. London: Africa Power and Politics Programme(APPP). Available at: http://www.dfid.gov.uk/r4d/PDF/Outputs/APPP/APPP-WP1.pdf (5 July 2011)

MacIntyre, A. (1971). Is a science of comparative politics possible? In A. MacIntyre(Ed.) Against the Self-Images of the Age: Essays on Ideology and Philosophy, (pp.260-79). London: Duckworth

Mahoney J., & and Goertz G. (2006). A tale of two cultures: Contrasting quantitativeand qualitative research. Political Analysis, 14(3), 227-249.

Mastenbroek, E., & Doorenspleet, R. (2007). Mind the gap! On the possibilities andpitfalls of mixed methods research. Paper presented at the 4th ECPR GeneralConference, September 6-8, Pisa, Italy. Available at:http://www.essex.ac.uk/ecpr/events/generalconference/pisa/papers/PP1466.pdf (23May 2011)

Meijer, P. Verloop, N., & Beijaard, D. (2001). Similarities and differences inteachers’ practical knowledge about teaching reading comprehension. The Journal ofEducational Research, 94 (3), 171-184.

Mak, L., & Marshall, S. (2004). Perceived mattering in young adults' romanticrelationships. Journal of Social and Personal Relationships, 21, 469-486.

Messick, S. (1989a). Meaning and values in test validation: The science and ethics ofassessment. Educational Researcher, 18(2), 5-11.

Messick, S. (1989b). Validity. In R. Linn (ed.) Educational Measurement (pp.13-103), New York: Macmillan.

Messick, S. (1995a). Standards of validity and the validity of standards inperformance assessment. Educational Measurement: Issues and Practice, 14 (4), 5-8.

Messick, S. (1995b). Validity of psychological assessment: Validation of inferencesfrom persons’ responses and performances as scientific inquiry into score meaning.American Psychologist, 50(9), 741-749.

Milton, J., Watkins, K., Studdard, S., & Burch, M. (2003). The ever widening gyre:Factors affecting change in adult education graduate programs in the United States.Adult Education Quarterly, 54(1), 23-41.

20

Moghaddam, F., Walker, B., & Harre, R. (2003). Cultural distance, levels ofabstraction, and the advantages of mixed methods. In A. Tashakkori & C. Teddlie(Eds.), Handbook of Mixed Methods in Social & Behavioral Research, (pp. 111-134).Thousand Oaks, CA: Sage

Moncagatta, P. (2009). The paradox of support for democracy in Venezuela. RECSMWorking Paper 4. Available at: http://www.upf.edu/survey/_pdf/RECSM_wp004.pdf(13 June).

Morse, J., & Niehaus, L. (2009). Mixed Method Design: Principles and Procedures.Walnut Grove, CA: Left Coast Press.

Munk, G., & Verkuilen, J. (2002). Conceptualizing and measuring democracy:Evaluating alternative indices. Comparative Political Studies, 35(1),5-34.

Myers, K., & Oetzel, J. (2003). Exploring the dimensions of organizationalassimilation: Creating and validating a measure. Communication Quarterly, 51, 438-457.

Nagle, R., Suldo, S., Christenson, S., & Hansen, A. (2004). Graduate students’perspectives of academic positions in school psychology. School PsychologyQuarterly, 19, 311-326.

Nassar-McMillan, S., Wyer, M., Oliver-Hoyo, M., & Ryder-Burge, A. (2010). Usingfocus groups in preliminary instrument development: Expected and unexpectedlessons learned. The Qualitative Report, 15(6), 1621-1634.

Northrop, F. (1947). The Logic of the Sciences and the Humanities. New York: WorldPublishing Company.

Onwuegbuzie, A., Bustamante, R., & Nelson, J. (2010). Mixed research as a tool fordeveloping quantitative instruments. Journal of Mixed Methods Research, 4(1) 56-78.

Onwuegbuzie, A. & Johnson, R. (2006). The validity issue in mixed research.Research in the Schools, 13(1), 48-63.

Oppenheim, A. (1992). Questionnaire Design, Interviewing and AttitudeMeasurement (2nd ed.). London, UK: Pinter.

Sartori, G. (1970). Concept misformation in comparative politics. American PoliticalScience Review, 64, 1033-1053.

Padgett, D. (1998). Qualitative Methods in Social Work Research: Challenges andRewards. Thousand Oaks, CA: Sage.

Pike, K. (1967). Language in Relation to a Unified Theory of the Structure of HumanBehavior. The Hague, Netherlands: Mouton.

21

Przeworski, A., Alvarez, M., Cheibub, J., & Limongi, F. (2000). Democracy andDevelopment: Political Institutions and Well-being in the World, 1950-1990.Cambridge, UK: Cambridge University Press.

Rowan, N., & Wulff, D. (2007). Using qualitative methods to inform scaledevelopment. The Qualitative Report, 12(3), 450-466.

Saris, W. & Gallhofer, I. (2007). Design, Evaluation, and Analysis of Questionnairesfor Survey Research. Hoboken, NJ: Wiley.

Sartori, G. (1994). Compare why and how: Comparing, miscomparing and thecomparative method. In M. Dogan & A. Kazancigil (Eds.), Comparing Nations:Concepts, Strategies, Substance, (pp. 14-34). London: Basil Blackwell.

Smith, P., Earp, J., & DeVellis R. (1995). Measuring battering: Development of theWomen’s Experience with Battering (WEB) scale. Women’s Health: Research onGender, Behavior, and Policy, 1(4), 273-288.

Tarrow, S. (1995). Bridging the quantitative-qualitative divide in political science.American Political Science Review, 89(2), 471-474.

Tashakkori, A., & Creswell, J. (2008). Mixed methodology across disciplines.Journal of Mixed Methods Research, 2(1), 3-6.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of Mixed Methods in Socialand Behavioral Research. Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2010). Sage Handbook of Mixed Methods in Socialand Behavioral Research (2nd ed.). Thousand Oaks, CA: Sage.

Tashiro, J. (2002). Exploring health promoting lifestyle behaviors of Japanese collegewomen: Perceptions, practices, and issues. Health Care for Women International,23(1), 59-70.

Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use ofmixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie(Eds.), Handbook of Mixed Methods in Social & Behavioral Research, (pp. 3-50).Thousand Oaks, CA: Sage.

Tomiyamal, T., Takeda, H., Yoshioka, M., & Shimomura, Y. (2003). Abduction forCreative Design (AAAI Tech. Rep. No. SS-03-02). Menlo Park, Ca: AAAI Press.

Van de Vijver, F. (1998). Towards a theory of bias and equivalence. ZUMANachrichten Spezial, Cross-cultural Survey Equivalence, 3, 41-65.

Van de Vijver, F. (2011). Bias and real differences in cross-cultural differences:Neither friends nor foes. In F. van de Vijver, A. Chasiotis, & S. Breugelmans (Eds.),Fundamental Questions in Cross-cultural Psychology, (pp. 235-257). Cambridge:Cambridge University Press.

22

Van de Vijver, F., & Leung, K. (1997). Methods and Data Analysis for Cross-Cultural Research. Thousand Oaks, CA : Sage.

Van de Vijver, F., & Tanzer, N. (2004). Bias and equivalence in cross-culturalassessment: An overview. European Review of Applied Psychology, 54(2), 119-135.

Viruel-Fuentes, E. (2007). Beyond acculturation: Immigration, discrimination, andhealth research among Mexicans in the United States. Social Science & Medicine, 65,1524-1535.

Vogt, D., King, D., & King L. (2004). Focus groups in psychological assessment:Enhancing content validity by consulting members of the target population.Psychological Assessment, 16, 231-243.