View
3.379
Download
3
Category
Tags:
Preview:
DESCRIPTION
Research Design and Validity. A series of six presentation, introduce scientific research in the areas of cross-cultural, using quantitative approach.
Citation preview
Quantitative Research Methodologies (2/6): Research Design and Validity
Prof. Dr. Hora Tjitra & Dr. He Quan
www.SinauOnline.com
2
Research Team
Research Program: Complex Problem Solving (CPS)
Hora Tjitra, HE Quan, LIU Manxia, SHI Wei, ZHANG Qi, YU
Zhaoyang, WANG Huiqi
Project Team Leader Complex Problem Solving Research project
Project Team MemberThe Police Bureau, Huzhou
Project Team Member Zhongcai Group, Hangzhou
Phd. Candidate, Zhejiang University Industry and Organization Psychology
Bachelor Degree, Zhejiang University Industry and Organization Psychology
3
Complex Problem Solving Stories
Economy
Environment
Technology
Education
Improve TheWell Being Of
Citizens in One Nation
Politics
Social
The Moral Issue aroused by the Biological Technology
Overexploiting The Underground Water in Beijing City
4
CPS Theoretical Framework
CPS Research Design
Observation System
• Group Interaction• Decision Making Process
Syntex System Data
• Corporate Results• Group Knowledge• Group Decisions
Questionnaire
• Quantitative Data
Interview Coaching
• Group Reflection
SYNTEX• Computer Business Simulation• Managing Complex Situations
Multi-Methods Approach
AmericaChina IndonesiaGermany France Japan
Cross-Cultural Perspective
6
Background Information and Experimental Process
Participants
Make decisions by themselves for company every month, for example the products order
and the material purchasing.
The Task
3-persons cooperate, 54 persons divided in 18 groups, the students coming from Zhejiang University
Program Display
7
Product Quality
The system structure behind Syntex business simulation
Material Storage
Purchasing Materials
Employee Satisfaction
Market Demand
HR Performance
Product Storage
Advertisement
Salary
Pricing Policy
Production Order
Recruitment Social Fund
Product Manufacture
Product Sales Company Income
Season / Weather
Competitor Price / Quality
Purchasing
+
-
- -
+
+
+
+
+
+
+
++
Employee Number
Oval represents the control variable, rectangle: system variable,
‘+’means active influence,’ –’mean negative influence .
Company Assets
Maintenance
Machine / Diesel
Leadership
HR Work Load
Lay-off Exchange
Resign
+
-
+-
+
+
+-
Number and Assortment
Sale PriorityMan.
Priority
and • Bank Account• Material Storage• Product Storage• Machine, Diesel
8
The differences between “best” and “worst” groups
The Performance=100*[2*Z(capital)+Z(number)+Z(satisfaction)]/4
Sorting the groups by score, selecting higher 8 groups as best groups, lower 8 groups as worst groups. ( t(14)=4.34, p=.001 )
Z=0Normal Distribution
The change of capital during 12 months:
Better groups can increase the capitalThe size of the company : better groups hire more people than worse
The change of employees’ satisfaction: both groups can’t manage very well, worse groups are worse
9
Information and Decision Making Difference
Best groupsBest groups Worst groupsWorst groups
Variables M SD M SD
Number of questions, total 124 29.76 123 20.71
Number of decisions, total 55 11.01 61 13.63
MANOVA( Repeated Measure)
Information Decision
Condition F(1/14)=.00 p=.95 F(1/14)=1.02 p=.33
Month F(5.69/79.72)=7.06 p=.00 F(8.51/69.18)=3.67 p=.00
Interaction F(5.69/79.72)=1.31 p=.26 F(8.51/69.18)=1.29 p=.25
10
The Analyzing Perspectives from the Data
AmericaChina IndonesiaGermany France Japan
3.Cross-Cultural Perspective
1.CPS Process Perspective
How do groups explore and manage the system? What is the difference between the best and worst groups in cognitive perspectives
2.Group Interaction Perspective
How do group member discuss and inquire the information?How do group member support each other?
11
Variables Used in Experimentation
Dependent Variable
Independent Variable
Variable
…One of the antecedent conditions manipulated by the experimenter
…The response of the organism; the variable that measures the influence the independent variable
…Any characteristic or phenomenon that can vary across organisms, situations, or environments
12
Three Main Research Designs Using Experimental Approach
A design in which the influence of extraneous variables is controlled for while the influence of the independent variable is tested
Research design: The outline, plan, or strategy used to investigate the research problem
A research design in which an experimental procedure is applied but all extraneous variables are not controlled
Experimental design
Quasi-Experimental
design
Single-case design
A research design in which a single groups of individuals is used to investigate the influence of a treatment condition
13
Experimental Design
• the experimental and the control groups‘ posttest scores are compared to assess the influence of the treatment condition
X
Y2
Y1
treatment measure
Posttest only design Pretest-posttest design
• the treatment effect is assesses by the difference between the experimental and control groups‘ pre- and posttest scores
X
Y2
Y1
treatment measurePremeasure
Y
Y
14
Quasi-Experimental Design
Nonequivalent comparison group design
Interrupted time-series design
• The results obtained from nonequivalent experimental and control groups are compared
Premeasure treatment measure
Y
Y
X Y1
Y2
• A treatment effect is assessed by comparing the pattern of pre- and posttest scores of one group of research participants
X Y
treatment measurePremeasure
Y Y Y
15
Basic Types of Designs for Case Studies
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
CONTEXT
Case
single-case designs multiple-case designs
holistic
(single-unit of
analysis)
CONTEXT
CaseEmbedded
Unit of Analysis 1
Embedded Unit of
Analysis 1
CONTEXTCase
CONTEXTCase
CONTEXTCase
CONTEXTCase
embedded
(multiple units of
analysis)
SOURCE: COSMOS Corporation
16
Research Validity
... the best available approximation to the truth or falsity of propositions, incl. propositions about cause
1. Is there a relationship between two variables ?
2. Given that there is a relationship, is it plausibly causal from one operational variables to the other or would the same relationship have been obtained in the absence of any treatments of any kind ?
3. Given that the relationship is plausibly causal and is reasonably known to be from one variable to another, what are the particular cause and effect constructs involved in the relationship ?
4. Given that there probably a causal relationship from construct A to construct B, how generalizable is this relationship across persons, settings, and times ?
17
Four Types of Validities
... Validity of the inference made about whether the independent and dependent variables co-vary
... Validity of the inference that independent and dependent variables are causally related
... Validity of the inference about the higher-order constructs from the operations used to represent them
... Validity of the inference about whether the causal relationship holds over people, setting, treatment variable, measurement variables, and time
Statistical Conclusion Validity
Internal Validity
Construct Validity
External Validity
18
Internal Validity
... a relationship between two variables is causal or that the absence of a relationship implies the absence of cause.
ACauses / Treatments /Independent Variables
BEffects / Outcomes /Dependent Variables
C2Comparison / Experimental /
Control Units
C1Extraneous Forces / Controlled Setting
1. Is there a relationship between A & B ?
2. Is there a causal relationship from A to B ?
3. Is there any other reason that could also caused B apart from A ?
19
Internal Validity - Threats
... Validity with which statements can be made about whether there is a causal relationship from one variable to another in the form in which the variables were manipulated or measured.
• History... an event which takes place between the pretest and posttest.
• Maturation... the respondent‘s growing older, wiser, stronger, more experienced, and the like between the pretest and posttest.
• Instrumentation... a change in the measuring instruments (e.g. observer learning effects)
• Testing ... Changes in person’s score on the
second administration of a test as a result of previously having taken the test
... is a threat when an observed effect might be due to ...
• Regression artifact ... The tendency for extreme scores to become less extreme on a second assessment
• Attrition ... Some people do not show up for the study or do not complete it
• Selection... The choice of participants for the various treatment groups is based on different criteria
• Additive and interactive effects... The combined effect of several threats to internal validity
20
... the approximate validity with which we can make generalizations about high-order constructs from research operations.
Construct Validity of Causes and Effects
AIndependent Variables
BEffects / Outcomes /Dependent Variables
Causes / Treatments
C2AIndependent Variables
C2BEffects / Outcomes /Dependent Variables
Manipulated Group 1
Control Group
C3AIndependent Variables
C3BEffects / Outcomes /Dependent Variables
Placebo
Manipulated Group 2
21
Construct Validity - Threats
• Inadequate Preoperational Explication of Constructs…the choice of operations should depend on the result of a conceptual analysis of the essential features of a construct (e.g. Aggression).
• Mono-Operation/ Method Bias…many experiments are designed to have only one exemplar of a particular possible cause, and some have just one measure to represent each of the possible effect constructs.
• Reactive Self-report Changes... it is possible that the treatments effects are due to the respondents being willing to present themselves to experimenters in ways that would lead to favorable evaluation.
• Experimenter Effects... which indicates that experimenter‘s attitudes/expectancies can bias the data obtained.
• Treatment Diffusion…individuals in one treatment group receive some or all of another group’s treatment
• Restricted Generalizability Across Constructs…sometimes treatments will affect dependent variables quiet differently, implying a positive effect on some construct and an unintended negative effect on another
22
... the presumed causal relationship can be generalized to and across alternate measures of the cause and effect and across different types of persons, settings, and times. (1)
External Validity
... the approximate validity with which conclusions are drawn about the generalizability of a causal relationship to and across populations of persons, settings, and times. (2)
1. Definition of the Target Populations(persons, settings, or times)
2. Samples to Represent These Populations
Types of Samples / Populations:
1. Target Populations
2. Formally Representative samples that correspondent to known populations
3. Sample Actually Achieved (in Field Research)
4. Achieved Populations
23
The categories of the external validity
... The extent to which the results of a study can be generalized to the larger population
Population Validity
... The extent to which the results of a study can be generalized across settings or environmental conditions
Ecological Validity
... The extent to which the results of a study can be generalized to the larger population
Temporal Validity
... The generallizability of results across different but related dependent variables
Outcome Validity
Treatment Variation Validity
... The generalizability of results across variation of the treatment
24
External Validity - Threats
• Interaction of Selection and TreatmentIn which categories of persons can a cause effect relationship be generalized?
• Interactions of Setting and TreatmentCan a causal relationship obtained in a factory be obtained in a military camp or an university campus?
• Interaction of History and TreatmentTo which period in the past and future can a particular causal relationship be generalized?
25
The literatures
Altschuler, L., Sussman,N.M., & Kachur, E. (2003). Assessing changes in intercultural sensitivity among physician trainees using the intercultural development inventory. International Journal of Intercultural Relations, 27(4), 387–401.
Leclerc, D., & Martin, J.N. (2004).Tour guide communication competence: French, German and American tourists' perceptions. International Journal of Intercultural Relations(4), 28,181-200.
Palthe, J. (2004).The relative importance of antecedents to cross-cultural adjustment: implications for managing a global workforce. International Journal of Intercultural Relations, 28(1), 37-59.
Strohschneider, S., & Guess, D. (1999). The Fate of the Moros: A Cross-cultural Exploration of Strategies in Complex and Dynamic Decision Making. International Journal of Psychology,34(4),235-252
@ Tjitra, 2010
Thanks YouAny comments & quest ions
are welcomeContact me at hora_t@sianuonline.com
26
www.SinauOnline.com
Recommended