Quantitative Research Methodologies (2/6): Research Design and Validity

Prof. Dr. Hora Tjitra & Dr. He Quan

www.SinauOnline.com

2

Research Team

Research Program: Complex Problem Solving (CPS)

Hora Tjitra, HE Quan, LIU Manxia, SHI Wei, ZHANG Qi, YU

Zhaoyang, WANG Huiqi

Project Team Leader Complex Problem Solving Research project

Project Team MemberThe Police Bureau, Huzhou

Project Team Member Zhongcai Group, Hangzhou

Phd. Candidate, Zhejiang University Industry and Organization Psychology

Bachelor Degree, Zhejiang University Industry and Organization Psychology

3

Complex Problem Solving Stories

Economy

Environment

Technology

Education

Improve TheWell Being Of

Citizens in One Nation

Politics

Social

The Moral Issue aroused by the Biological Technology

Overexploiting The Underground Water in Beijing City

4

CPS Theoretical Framework

CPS Research Design

Observation System

• Group Interaction• Decision Making Process

Syntex System Data

• Corporate Results• Group Knowledge• Group Decisions

Questionnaire

• Quantitative Data

Interview Coaching

• Group Reflection

SYNTEX• Computer Business Simulation• Managing Complex Situations

Multi-Methods Approach

AmericaChina IndonesiaGermany France Japan

Cross-Cultural Perspective

6

Background Information and Experimental Process

Participants

Make decisions by themselves for company every month, for example the products order

and the material purchasing.

The Task

3-persons cooperate, 54 persons divided in 18 groups, the students coming from Zhejiang University

Program Display

7

Product Quality

The system structure behind Syntex business simulation

Material Storage

Purchasing Materials

Employee Satisfaction

Market Demand

HR Performance

Product Storage

Advertisement

Salary

Pricing Policy

Production Order

Recruitment Social Fund

Product Manufacture

Product Sales Company Income

Season / Weather

Competitor Price / Quality

Purchasing

+

-

- -

+

+

+

+

+

+

+

++

Employee Number

Oval represents the control variable, rectangle: system variable,

‘+’means active influence,’ –’mean negative influence .

Company Assets

Maintenance

Machine / Diesel

Leadership

HR Work Load

Lay-off Exchange

Resign

+

-

+-

+

+

+-

Number and Assortment

Sale PriorityMan.

Priority

and • Bank Account• Material Storage• Product Storage• Machine, Diesel

8

The differences between “best” and “worst” groups

The Performance＝100*[2*Z（capital）+Z（number）+Z（satisfaction）]/4

Sorting the groups by score, selecting higher 8 groups as best groups, lower 8 groups as worst groups. ( t(14)=4.34, p=.001 )

Z=0Normal Distribution

The change of capital during 12 months:

Better groups can increase the capitalThe size of the company : better groups hire more people than worse

The change of employees’ satisfaction: both groups can’t manage very well, worse groups are worse

9

Information and Decision Making Difference

Best groupsBest groups Worst groupsWorst groups

Variables M SD M SD

Number of questions, total 124 29.76 123 20.71

Number of decisions, total 55 11.01 61 13.63

MANOVA( Repeated Measure)

Information Decision

Condition F(1/14)=.00 p=.95 F(1/14)=1.02 p=.33

Month F(5.69/79.72)=7.06 p=.00 F(8.51/69.18)=3.67 p=.00

Interaction F(5.69/79.72)=1.31 p=.26 F(8.51/69.18)=1.29 p=.25

10

The Analyzing Perspectives from the Data

AmericaChina IndonesiaGermany France Japan

3.Cross-Cultural Perspective

1.CPS Process Perspective

How do groups explore and manage the system? What is the difference between the best and worst groups in cognitive perspectives

2.Group Interaction Perspective

How do group member discuss and inquire the information?How do group member support each other?

11

Variables Used in Experimentation

Dependent Variable

Independent Variable

Variable

…One of the antecedent conditions manipulated by the experimenter

…The response of the organism; the variable that measures the influence the independent variable

…Any characteristic or phenomenon that can vary across organisms, situations, or environments

12

Three Main Research Designs Using Experimental Approach

A design in which the influence of extraneous variables is controlled for while the influence of the independent variable is tested

Research design: The outline, plan, or strategy used to investigate the research problem

A research design in which an experimental procedure is applied but all extraneous variables are not controlled

Experimental design

Quasi-Experimental

design

Single-case design

A research design in which a single groups of individuals is used to investigate the influence of a treatment condition

13

Experimental Design

• the experimental and the control groups‘ posttest scores are compared to assess the influence of the treatment condition

X

Y2

Y1

treatment measure

Posttest only design Pretest-posttest design

• the treatment effect is assesses by the difference between the experimental and control groups‘ pre- and posttest scores

X

Y2

Y1

treatment measurePremeasure

Y

Y

14

Quasi-Experimental Design

Nonequivalent comparison group design

Interrupted time-series design

• The results obtained from nonequivalent experimental and control groups are compared

Premeasure treatment measure

Y

Y

X Y1

Y2

• A treatment effect is assessed by comparing the pattern of pre- and posttest scores of one group of research participants

X Y

treatment measurePremeasure

Y Y Y

15

Basic Types of Designs for Case Studies

CONTEXT

Case

CONTEXT

Case

CONTEXT

Case

CONTEXT

Case

CONTEXT

Case

single-case designs multiple-case designs

holistic

(single-unit of

analysis)

CONTEXT

CaseEmbedded

Unit of Analysis 1

Embedded Unit of

Analysis 1

CONTEXTCase

CONTEXTCase

CONTEXTCase

CONTEXTCase

embedded

(multiple units of

analysis)

SOURCE: COSMOS Corporation

16

Research Validity

... the best available approximation to the truth or falsity of propositions, incl. propositions about cause

1. Is there a relationship between two variables ?

2. Given that there is a relationship, is it plausibly causal from one operational variables to the other or would the same relationship have been obtained in the absence of any treatments of any kind ?

3. Given that the relationship is plausibly causal and is reasonably known to be from one variable to another, what are the particular cause and effect constructs involved in the relationship ?

4. Given that there probably a causal relationship from construct A to construct B, how generalizable is this relationship across persons, settings, and times ?

17

Four Types of Validities

... Validity of the inference made about whether the independent and dependent variables co-vary

... Validity of the inference that independent and dependent variables are causally related

... Validity of the inference about the higher-order constructs from the operations used to represent them

... Validity of the inference about whether the causal relationship holds over people, setting, treatment variable, measurement variables, and time

Statistical Conclusion Validity

Internal Validity

Construct Validity

External Validity

18

Internal Validity

... a relationship between two variables is causal or that the absence of a relationship implies the absence of cause.

ACauses / Treatments /Independent Variables

BEffects / Outcomes /Dependent Variables

C2Comparison / Experimental /

Control Units

C1Extraneous Forces / Controlled Setting

1. Is there a relationship between A & B ?

2. Is there a causal relationship from A to B ?

3. Is there any other reason that could also caused B apart from A ?

19

Internal Validity - Threats

... Validity with which statements can be made about whether there is a causal relationship from one variable to another in the form in which the variables were manipulated or measured.

• History... an event which takes place between the pretest and posttest.

• Maturation... the respondent‘s growing older, wiser, stronger, more experienced, and the like between the pretest and posttest.

• Instrumentation... a change in the measuring instruments (e.g. observer learning effects)

• Testing ... Changes in person’s score on the

second administration of a test as a result of previously having taken the test

... is a threat when an observed effect might be due to ...

• Regression artifact ... The tendency for extreme scores to become less extreme on a second assessment

• Attrition ... Some people do not show up for the study or do not complete it

• Selection... The choice of participants for the various treatment groups is based on different criteria

• Additive and interactive effects... The combined effect of several threats to internal validity

20

... the approximate validity with which we can make generalizations about high-order constructs from research operations.

Construct Validity of Causes and Effects

AIndependent Variables

BEffects / Outcomes /Dependent Variables

Causes / Treatments

C2AIndependent Variables

C2BEffects / Outcomes /Dependent Variables

Manipulated Group 1

Control Group

C3AIndependent Variables

C3BEffects / Outcomes /Dependent Variables

Placebo

Manipulated Group 2

21

Construct Validity - Threats

• Inadequate Preoperational Explication of Constructs…the choice of operations should depend on the result of a conceptual analysis of the essential features of a construct (e.g. Aggression).

• Mono-Operation/ Method Bias…many experiments are designed to have only one exemplar of a particular possible cause, and some have just one measure to represent each of the possible effect constructs.

• Reactive Self-report Changes... it is possible that the treatments effects are due to the respondents being willing to present themselves to experimenters in ways that would lead to favorable evaluation.

• Experimenter Effects... which indicates that experimenter‘s attitudes/expectancies can bias the data obtained.

• Treatment Diffusion…individuals in one treatment group receive some or all of another group’s treatment

• Restricted Generalizability Across Constructs…sometimes treatments will affect dependent variables quiet differently, implying a positive effect on some construct and an unintended negative effect on another

22

... the presumed causal relationship can be generalized to and across alternate measures of the cause and effect and across different types of persons, settings, and times. (1)

External Validity

... the approximate validity with which conclusions are drawn about the generalizability of a causal relationship to and across populations of persons, settings, and times. (2)

1. Definition of the Target Populations(persons, settings, or times)

2. Samples to Represent These Populations

Types of Samples / Populations:

1. Target Populations

2. Formally Representative samples that correspondent to known populations

3. Sample Actually Achieved (in Field Research)

4. Achieved Populations

23

The categories of the external validity

... The extent to which the results of a study can be generalized to the larger population

Population Validity

... The extent to which the results of a study can be generalized across settings or environmental conditions

Ecological Validity

... The extent to which the results of a study can be generalized to the larger population

Temporal Validity

... The generallizability of results across different but related dependent variables

Outcome Validity

Treatment Variation Validity

... The generalizability of results across variation of the treatment

24

External Validity - Threats

• Interaction of Selection and TreatmentIn which categories of persons can a cause effect relationship be generalized?

• Interactions of Setting and TreatmentCan a causal relationship obtained in a factory be obtained in a military camp or an university campus?

• Interaction of History and TreatmentTo which period in the past and future can a particular causal relationship be generalized?

25

The literatures

Altschuler, L., Sussman,N.M., & Kachur, E. (2003). Assessing changes in intercultural sensitivity among physician trainees using the intercultural development inventory. International Journal of Intercultural Relations, 27(4), 387–401.

Leclerc, D., & Martin, J.N. (2004).Tour guide communication competence: French, German and American tourists' perceptions. International Journal of Intercultural Relations(4), 28,181-200.

Palthe, J. (2004).The relative importance of antecedents to cross-cultural adjustment: implications for managing a global workforce. International Journal of Intercultural Relations, 28(1), 37-59.

Strohschneider, S., & Guess, D. (1999). The Fate of the Moros: A Cross-cultural Exploration of Strategies in Complex and Dynamic Decision Making. International Journal of Psychology,34(4),235-252

@ Tjitra, 2010

Thanks YouAny comments & quest ions

are welcomeContact me at hora_t@sianuonline.com

26

www.SinauOnline.com