Quantitative Research: Surveys and Experiments

IS541, Lecture 6a: “Quantitative Research: Surveys and Experiments”

Master Management, Master Business Informatics; March 4th, 2015

Martin KretzerChair of Information Systems IV, Business School and

Institute for Enterprise Systems (InES), University of Mannheim

Overall Course Structure

#9 Final

Assignm

ent#5a Literature Review Intro

#5b Literature Review RBD

2ManTIS FSS 2015 - Quantitative

Research

#7a Design Science Intro

#7b Design Science RBD

#8a Qualitative Research Intro

#8b Qualitative Research RBD

#6a Quantitative Research Intro

#6b Quantitative Research RBD

#1 Introduction

#2 Theories

#3 Methods

#4 Scientific Writing

and Publishing

Goals of this Lecture

A

Understand the basics of survey-

based research and experiments

Know the research process of

surveys and experiments

Data gathering and analysis

Be aware of important quality

criteria for quantitative research

Learn best practices

Get to know popular software tools

for analyzing quantitative data3ManTIS FSS 2015 - Quantitative

Research

Agenda

Agenda

1 Basics of Quantitative Research

2 Surveys

3 Experiments

4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)

5 Summary

6 Supplementary Material on Quality Criteria

7 References

4ManTIS FSS 2015 - Quantitative Research

The Two Main Paradigms of Empirical Work

Quantitative Qualitative

Mostly deductive (theory first) Mostly deductive (observation first)

Statistical generalizability Analytical generalizability

Linear, pre-planned research design Evolving, iterative research design

High number of observations Focused number of observations

Statistical analyses Conceptual analyses

Independent of context Context-dependent

Reliability is key Authenticity is key

Source: Denzin and Lincoln (2011), Neumann (2000)

After spring breakToday


Research

Characteristics of Quantitative Research

Strongly connected to a positivist epistemological stance ( lecture 2)

– Objective reality can be captured and translated into testable hypotheses

– Researcher can capture empirical data that allows them to make inferences about that reality

Generally emphasizes high “n”

– Numbers represent values and levels of theoretical constructs and concepts

– Interpretation of the numbers is viewed as strong scientific evidence of how a phenomenon works

– Aims for statistical generalizability to make predictions among unobserved members of the

population

Strongly relies on statistical tools as an essential element in the researcher's toolkit

6

Source: Straub et al. (2005)

ManTIS FSS 2015 - Quantitative

Research

Quantitative Research Methods (revisited)

Simulation Field Experiment / Quasi Experiment

State a hypothesis

Imitate some real process or action to prove your hypothesis

Suitable for observing correlation between variables

Strengths: allows estimation and prediction

Conducted in field settings, e.g. real organization

Rare, because of the difficulties associated with manipulating

treatments and controlling for extraneous effects in a field

setting

Strengths: high internal and external validity

Survey Laboratory Experiment

Collect self reported data of people

Standardized questionnaire or interview

Suited to study preferences, thoughts, and behavior of

people

Suited for descriptive, exploratory or explanatory research

Strengths: collect unobservable data (thoughts); remote

collection; convenient for respondents; can detect small

effects

Independent variables are manipulated by the researcher (as

treatments)

Subjects are randomly assigned to treatment

Results of the treatments are observed

Suited for explanatory research to examine individual cause-

effect relationships in detail

Strengths: influence of individual factors can be well

explained; very high internal validity (causality)

Source: Bhattacherjee (2012), Myers (2009)


Research

Examples for Quantitative Studies

8

Survey

– Goal: Identify drivers for user satisfaction

– Build a questionnaire for (1) measuring satisfaction with a new IS and (2) measuring the constructs that you expect could influence satisfaction.

– Distribute the questionnaire to your sample (e.g. people in your organization)

– Analyze answered questionnaires by analyzing which drivers can be associated with satisfaction (e.g., through computing correlation)

Experiment

– Goal: Examine the effects of your new online shop

– Build your treatment groups, e.g.:

• Online shop that recommends items

• Online shop that does not recommend items

– Randomly assign individuals to a treatment group

– After 4 weeks check whether your recommender increased sales

Simulation

– Goal: Examine the effect of a new algorithm

– Build your treatment groups (e.g., old algorithm, new algorithm) and check for differences

– Can be referred to as „computational experiment“


Research

conceptual

observable

unobservable

Theory’s Basic Constituents and Mechanisms

Source: Bhattacherjee (2012, p. 39)

Empirical

Plane

Theoretical

Plane Construct A Construct BProposition

Independent

Variable

Dependent

VariableHypothesis

External validity

Internal validity


Research

Classes of Variables

10

IntelligenceAcademic

Achievement

Earning

Potential

Effort

Independent

Variable

Moderating

Variable

Mediating

Variable

Dependent

Variable



Research

Internal Validity and External Validity

Internal validity External validity

= Causality

Does a change in X really cause a change in Y?

Three conditions

1. Covariation of cause and effect

2. Temporal precedence

3. No plausible alternative explanation

Note: causality <> correlation!

= Generalizability

Can the observed association be generalized from

sample to the population or further contexts?



Research

Internal Validity and External Validity

12Source: Bhattacherjee (2012, p. 38)

•There is no single „best“ research method!•You need to know the strengths and limitations of your research method!•Combinations might make sense (also mixing quantitative and qualitative research methods; see Venkatesh et al. 2013 for further information)


Research

Internal Validity (Causality)

Why is internal validity of cross-sectional field surveys typically limited?

– Independent variable Y cannot be manipulated

– Cause and effect are measured at the same time you do not know whether X

causes Y or Y causes X

Why is internal validity of laboratory experiments typically high?

– Independent variable Y can be manipulated via a treatment

– Effect can be observed after a certain point in time

– External factors can be controlled


Research

External Validity (Generalizability)

Why is external validity of laboratory experiments typically limited?– Artifical treatments

– External factors are controlled But: in real settings external factors cannot be controlled!

Why is external validity of cross-sectional surveys typically high?– Data from a wide variety of individuals or firms is collected

Qualitative research: Why may single case studies have higher generalizability than multiple case studies?– In qualitative research studies, you have to clearly describe the context of your

study

– The more detailed you can describe the context, the better you can explain to which further cases your results can be generalized!


Research

conceptual

observable

unobservable

Theory’s Basic Constituents and Mechanisms (cont’d)


Empirical

Plane

Theoretical

Plane Construct A Construct BProposition

Independent

Variable

Dependent

VariableHypothesis

External validity

• Internal validity

• Statistical conclusion validity

Construct

validity

Construct

validity


Research

Construct validity and statistical conclusion validity

Construct validity Statistical conclusion validity

How well is our measurement scale measuring the

independent variable and the theoretical construct

that it is expected to measure?

Are the statistical conclusions really valid?

Did we select the right statistical method for testing

the hypothesis?

Does the sample meet the requirements?

(only relevant for quantitative research)



Research

Agenda

Agenda


2 Surveys

3 Experiments


5 Summary


7 References


Using surveys: advantages and disadvantages

Biases

Non-response bias

Common method bias

Sampling bias

Social desirability bias

Flexible and Efficient High Volume Data Analysis

Application across all research

phases

Measure unobservable data

(Preferences, Attitudes, Traits)

Economical in terms of

researcher time, effort and cost

Can be administered to a

high number of subjects

Remote data collection

Comparability through

standardization

Well established quality

criteria

Statistical tests

Detect small effects with

large samples

Richness of Data

Only answers to standardized questions

Interpretation and context of respondent

missing

Source: Bhattacherjee (2012)


Research

Structural and measurement model

19

Structural model defines abstract relationship between constructs

Measurement model contains empirically observed variables

Legend:

latent exogenous variable

latent endogenous variable

x measurable exogenous indicator

y measurable endogenous indicator

path coefficient between latent exogenous and

endogenous variables

path coefficient between latent variables and

measurable indicators

measurement error

y2y1

3

Measurement Model (Outer Model)

Structural Model (Inner Model)

3 4

4

x2x1

1

1 2

2

Source: Williams et al. (2009)


Research

Attention: List of terms that are frequently used

interchangeably in positivistic reseach!

Measurable indicator (e.g., x1, x2, x3, y1, y2, y3)

– „Question“

– Indicator

– Item

– Measure

– Measurable variable you should not use this term!

Independent variable (IV) (e.g., x)

– (Latent) Exogenous variable

– Factor (typically refers to an IV that uses a nominal scale in experiments)

– Antecedent

– Cause

Dependent variable (DV) (e.g., y)

– (Latent) Endogenous variable

– Outcome

– Effect

Moderation effect

– Interaction effect


Research

Scale development for measurement indicators

21

Definition Scaling Process: Process of developing scales to measure indicators (items)

Rating scales are attached to items (unidimensional or multidimensional scaling)

Scales define what the respondent can choose to answer

Depending on the chosen scale certain statistic analysis are possible

Description Example

Equ

al A

ppea

ring

Participants rate items with

“agree” or “disagree”

Items only “appear” equal; in fact they represent

different values for measuring a certain concept

Thurstone

agree disagree

I like doing sports. X

I like swimming. X

Sum

mat

ive

/ Cum

ulat

ive

Likert

Participants rate items on a 5-point or

7-point scale

Scale ranges from “strongly agree” to “strongly

disagree”

strongly strongly

agree … neutral … disagree

I like …. X

I like…. X

Guttman

Goal: Cumulative scale

Participants rate items with “yes”/“no”

Creates a sorted matrix or table (see example)

yes no

Do you mind immigrants in your city? X

Would you live next to an immigrant? X

Would you marry an immigrant? X

Source: Bhattacherjee (2012, pp. 48-50)

ManTIS FSS 2015 - Quantitative Research

Number of Items per Construct

Why is the number of items for measuring a variable so important?

– Problem with only one item: risk of random error may be high

– Problem with too many items: people will stop answering your survey

How many questions ( items) should I ask?

– Almost no risk for random erorr: 1 item

– Stable constructs: 3-6 items

– Rather new constructs: at least 8 items

Examples:

– Age 1 item

– Perceived ease of use (Davis 1989):

• First pretest: 16 items

• Final items: 6 items

– Perceived ease of use (today): usually 3 items


Research

Quality Criteria: Measurement Model Validity

Summary of quality criteria for the measurement model


Research

Quality criterion Recommendation

Convergent validity of a

single indicator

High average factor loadings: λ > 0.7

Narrow range of factor loadings: λmax - λmin < 0.2


single construct

High average variance extracted: AVE > 0.5

High composite reliability: ρc > 0.8

High communality index of the construct

Convergent validity of the

measurement modelHigh average communality index

Discriminant validity of a

single indicatorEach item loading is greater than all cross-loadings


single construct

A construct’s AVE is greater than the squared construct’s

correlation with any other construct

No common method bias

Substantive factor loadings are greater than method factor

loadings

Method factor loadings are not significant while

substantive factor loadings are significant

(Details in section 6 “Supplementary Material on Quality Criteria”)

Quality Criteria: Structural Model Validity


Research

Summary of quality criteria for the structural model (PLS regression analysis)


Direct effectHigh standardized path estimates

Bootstrap algorithm and t-test

Moderating effect ANOVA and F-test or product-indicator approach in conjunction with PLS

Predicting power

High variance explained (R² > 0.2)

High effect size 𝑓²: 0.02 small effect; 0.15 medium effect; 0.35 large effect

High redundancy

Global quality of structural

modelHigh goodness of fit

No multicollinearity

No perfect correlation between independent variables: Standardized path

estimates < 0.8

High tolerance of independent variables: tolerance > 0.1

Small variance inflation factor of independent variables: VIF < 10

(Details in section 6 “Supplementary Material on Quality Criteria”)

Golden Circle Analysis – Survey Example

Golden Circle Analysis (GCA)

“Privacy Concerns and Privacy-Protective

Behavior in Synchronous Online Social

Interactions”

– Authors: Z.J. Jiang, C.S. Heng, B.C.F. Choi

– Year: 2013

– Outlet: Information Systems Research (ISR)

• Vol. 24, No. 3, pp. 579-595


Research

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis

GCA Survey: Brief Summary

Individuals‘ behavior is at times inconsistent with their privacy

concerns (e.g., they disclose private information in synchronous

online social interaction although they know the risks)

Focus: privacy concerns versus social rewards

Students conduct 3 chatroom sessions afterwards a survey is

administered

Findings:

– Individuals use self-disclosure and misrepresentation to protect their privacy

– Social rewards explain deviations

26

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Motivation

Privacy trade-off in the context of online social interactions

– Online social interactions may generatie multiple benefits:

synchronous exchange of information, sharing of cultural artifacts,

self-presentation, feedback from peers, socio-emotional support,

a borderless „space“

– However, 33% of internet users are concerned about their privacy

in online social interactions ( the paper lists numerous threats)

– Ironically, many users are still likely to disclose private information

even if they become aware of the risks

27

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Research Question

28

The paper clearly states a research question:

– Why is users‘ privacy behavior at times inconsistent with their

privacy concerns?

– P. 580, end of second paragraph

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Research Objective

29

The paper provides a clear definition of its research

objectives

– P. 580, last paragraph

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Theory Base

Paper is not based on a single, specific theory

However, the paper integrates two research streams to build a research model:

– Hyperpersonal framework• Approach for understanding how users of mediated communications experience relational

intimacy

– Privacy calculus • Relational privacy trade-off: privacy concerns versus rewards from disclosing private

information

Research Model:

30

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Research Design – Data Collection

Pilot: 3 rounds of preliminary tests to compare and evaluate different

methods of data collection (Appendix A)

Research Design

– Sample: 251 students in Singapore

– Three online chat sessions, each lasting 1 hour

Survey

– Survey after the end of the third chat session

– All items measured on a 7-point Likert scale ranging from 1 „strongly

disagree“ to 7 „strongly agree“. E.g., privacy concerns awareness:

31

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Research Design – Data Analysis

Data Analysis:

– Partial least squares (PLS) regression

– Measurement model assessment:

• Individual item reliability

• Internal consistency

• Discriminant validity

• Item loadings, cross-loadings, composite reliability, average variance

extracted (AVE)

– Structural model assessment

• Correlations

• Path coefficients and hypotheses testing all hypotheses were confirmed

• R²

• Sobel tests to examine whether privacy concerns and social rewards fully

mediate the effects

• Confirmatory factor analysis (CFA) for two models in order to test for

common method bias

32

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research


Measurement model assessment:

33

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research


Measurement model and structural model assessment:

34

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research


Structural model assessment:

35

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Contribution

Extension of the privacy calculus perspective to the context of

synchronous online social interactions

– This contribution is valuable, because past research „has

predominantly applied the privacy calculus to commercial contexts“ (p.

590)

– „In the absence of monetary or tangible rewards, social rewards are

just as attractive in balancing privacy concerns and governing

individuals‘ behavior.“ (p. 590)

Identification of four antecedents (hyperpersonal framework) of

privacy concerns and social rewards

Disclosure and nondisclosre are not the only two possible actions

stemming from privacy protection misrepresentation is a third

action (and independent from the established two actions)

Explanation of the different roles „anonymity of self“ and

„anonymity of others“ in online social interactions

36

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Analysis of Links

Practical motivation results in the research question (RQ)

Theoretical motivation builds on RQ and results in three

clearly stated research objectives

To address the research objectives, two research streams

are reviewed and integrated into one model

A survey for confirming the model seems to be an excellent

choice

The contribution section (section 6.2) summarizes the survey

results and explains how they extend existing literature.

Thereby, it directly addresses the RQ and the motivation of

the paper ( “golden circle”)

37

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Survey: Summary of Analysis

All elements of the GCA are addressed in the paper

The reader can easily follow the central theme („Roter

Faden“)

Contribution seems valuable

10 Hypotheses are confirmed and typical quality criteria for

surveys is met

Limitations and future research directions are also oulined

38

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

Agenda

Agenda


2 Surveys

3 Experiments


5 Summary


7 References


Experiment Designs

Between-subjects design

– = Participants can be part of only one treatment group (and are then compared to the „control

group“)

– Advantage: no carryover effects

Within-subjects design

– = Every single participant is subject to each treatment (incl. The control)

– Advantage: statistical significance

Mixed design

– E.g., between-subjects design for independent variable A and within-subjects design for

independent variable B

– Example: Mixed experiment design of Master thesis on the next few slides


Research

Mixed Design Experiment Example: Model

41

Perceived ease of use

Expertise

Factor 1

Business Intelligence

Client

Factor 2

Report Recommendation

Factor 3

Potential interaction effects:

• Factor 1 * Factor 2



• Factor 1 * Factor 2 * Factor 3

• 7 effects that should be tested!

Design:

• Factor 1 (Expertise): within-subjects variable

• Factor 2 (BI Client): within-subjects variable

• Factor 3 (Rep. Rec.): between subjects variable


Research

Mixed Design Experiment Example: Design Introduction

User Expertise Ease of Use of

SAP

BusinessObjects

Ease of Use of

Microsoft Excel

Ease of Use of

Tableau Desktop

Perceived ease of use

of recommendation

Counterbalanced

Counterbalanced

Factor 2: Business Intelligence Client

Factor 3: Recommender

Factor 1:

User Expertise


Research

Mixed Design Experiment Example: ANOVA Results

Statistical analysis of multiple factors (i.e., nominally scaled independent

variables) on one independent variable mostly Analysis of Variance

(ANOVA )

If you have multiple independent variables Multiple ANOVA

(MANOVA)

43

Df Sum Sq Mean Sq F value P(>F)

Between-subjects:

EXP 4 0.42 0.106 0.048 0.995

CLIENT 2 6.93 3.464 1.587 0.227

EXP*CLIENT 8 24.23 3.029 1.387 0.256

Residuals 22 48.03 2.183

Within-subjects:

REC 1 4.879 4.879 4.010 0.058+

REC*EXP 4 14.629 3.657 3.006 0.040*

REC*CLIENT 2 0.496 0.248 0.204 0.817

REC*EXP*CLIENT 8 13.952 1.744 1.433 0.238

Residuals 22 26.769 1.217

Dependent variable: perceived ease of use; n=37. Significance: *p<0.05; +p<0.10


Research

Mixed Design Experiment Example: Graph. Results

44

(M)ANOVA only indicates effects but no directions of the effect!

Thus, you need to draw the effect and interpret the figure!

Note: If the lines in your graphic are parallel, then there is no interaction effect at all!


Research

Common ways for increasing internal validity of experiments

• Manipulate independent variables in one or more levels (treatment)

• Compare the effects of the treatments against a control group

• In experimental designs subjects must recognize different treatmentsTreatment

• Eliminate extraneous variables by holding them constant

• For example restricting a study to a single genderElimination

• Consider additional extraneous variables

• Separately estimate their effects on the dependent variable (e.g., via factorial designs where one factor is gender)

Inclusion

• Measure extraneous variables

• Use them as covariates during the statistical testing processStatistical control

• Cancel out effects of extraneous variables through a process of random sampling (if random nature is proven)

• Two types: random selection, random assignmentRandomization

• Randomize the order of experimental treatments

• Reduces error due to carryover effectsCounterbalance

45

Source: Bhattacherjee (2012, pp. 39-40)


Research

Golden Circle Analysis – Experiment Example

Golden Circle Analysis (GCA)

“The Nature and Consequences of Trade-Off

Transparency in the Context of

Recommendation Agents”

– Authors: J.D. Xu, I. Benbasat, R.T. Confetelli

– Year: 2014

– Outlet: Management Information Systems Quarterly

(MISQ)

• Vol. 38, No. 2, pp. 379-406


Research

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis

GCA Experiment: Brief Summary

The authors investigate the impact of a novel design

feature for a recommendation agent (RA): trade-off

transparency (TOT)

The TOT design feature directly influences „consumer‘s perceived

product diagnosticity“ and „perceived enjoyment“

The authors find that there exists an optimal maximum in TOT

Furthermore, the authors identify diagnosticity and enjoyment as two

antecedents for „perceived decision quality“ and „perceived decision

effort“

47

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Motivation

The authors reference four sources which indicate the

economic necessity of RAs for online shops. But: poorly

designed RAs have negative effects!

Influence of specific design attributes of RAs on decision

making and other outcomes is still not well understood

Overall many sources that indicate benefit and importance

of RAs

RAs ability to capture consumer‘s product attribute

preferences is identified as a „central function of RAs“

Explanation of potential benefits that might arise if users

have better knowledge about pros and cons of different

laptops when browsing an online shop

48

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Motivation

Explicit identification of a research gap that needs to be

filled:

– P. 380, last sentence of third paragraph

49

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Research Question

50

No specific research question statedBrief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Research Objective

51

Three research objectives are stated specifically

1. Examine the impact of a trade-off transparent RA on perceived

enjoyment and perceived product diagnosticity. The context are

laptops in an online shop. Example trade-off: price vs. hard-drive

capacity; weight vs. screen size

2. Examine whether there is an optimal maximum of TOT

3. Extend and challenge the effort-accuracy framework because the RA

enables more accurate decisions to be made without simultaneously

increasing efforts

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Theory Base

Stimulus-Organism-Response (S-O-R) model

– Adopted from marketing and psychology research

– External cues (e.g., design features of online shops) influence a consumer‘s affective and/or

cognitive processes; which in turn determine the consumer‘s behavioral and/or internal

response

– Overarching framework for the authors‘ own theoretical model Operationalization:

• Stimulus: trade-off transparency feature of an online RA

• Organism: the user‘s enjoyment (affective system), perceived product diagnosticity

(cognitive system)

• Response: the user‘s perceived decision quality and perceived decision effort

Cognitive load theory

– TOT improves decisions up to a certian point. After that point, TOT overburdens users‘

cognitive limitations and is counterproductive

Effort-Accuracy framework

– An increase in decision accuracy is accompanied by an increasing in the decision makers

efforts (the „longstanding effort-accuracy conflict“)

52

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Theory Base

Proposed Theoretical Model– Enjoyment an affective reaction (i.e., an emotional response when interacting with a stimulus)

– Product diagnosticity = extent to which a consumer believes that a system is helpful for fully evaluating a product a cognitive reaction (i.e., a user‘s mental process when interacting with the stimulus)

Based on their proposed theoretical model, the authors develop 10 hypotheses– Note: The authors hypothesize inverted U-shaped curves as the level of trade-off increases on

perceived enjoyment (H3) and perceived product diagnosticity (H4) --> assumption: an optimal maximum exists!

– Research model on next few slides

53

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Research Design – Data Collection

Implementation of design feature

– Horizontal scales with „slider“ represent the value of each product attribute

– If the user moves one slider, other sliders will automatically be moved, too the

user can directly observe the trade-off dependencies between several attributes

– The greater the TOT, the more sliders are moved automatically

– In total, the online shop offers 50 laptops

54

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research


160 participants (of which 131 are undergraduate students)

Treatment groups:

– „Self-Shoppers“ versus „Friend Shoppers“• 50% of participants are asked to shop for themselves: if participants are

shopping for themselves, their initial product preference can be compared with their final attribute preferences

• 50% of participants are asked to shop for a fictitious fried: prior research indicates that shopping for friends helps minimize the effects of negative emotions when making attribute trade-offs

– Trade-off transparency• Low (25% of participants)

• Medium (25% of participants)

• High (25% of participants)

• Control no specific trade-off transparency (25% of participants)

Between-subjects design – Each participant is assigned to one of the 2*4 = 8 treatment groups

– 160 / 8 = 20 participants per group a power analysis test indicates sufficient statistical power (0.80) to detect a medium effect size (f=0.25)

55

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research


Instructions for participants

Experimental procedure

– Questionnaire related to demographic and control variables website training

random assignment to a group Questionnaire related to DVs

56

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

Instruction for “Self-Shoppers” Instruction for “Friend Shoppers”

After the instruction, the user selects a

value range (in USD) for eight laptop

attributes

GCA Experiment: Research Design – Data Analysis

Statistical analysis

– MANOVA for testing the effects of trade-off transparency

– PLS regression for testing

• All items from previous literature (p. 391, tbl. 4)

57

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research


Manipulation check

– Numbers of shown trade-offs was measured

– Users‘ awareness of trade-offs was measured

Effect of Trade-Off Transparency levels

– MANOVA (incl. Pillari‘s trace, Wilk‘s lambda, Hotelling‘s trace, Roy‘s largest

root) results are significant further ANOVAs on the two DV‘s separately

– Product diagnosticity• s

58

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research


PLS results

– Measurement model assessment: loadings and cross loadings

– Structural model assessment: composite reliability, Cronbach‘s alpha, AVE,

path coefficients, R²

– Since this GCA focuses on the paper as an exemplary experiment paper, the

PLS results of the questionnaire are not presented in detail for PLS

analysis, please have a look at the survey GCA

59

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Contribution

Results– Trade-off transparent RA improves perceived enjoyment and perceived product

diagnosticity

– Medium level of TOT has the best effect

– Besides H8, all hypotheses are confirmed

TOT feature helps to identify how users‘ attribute choices are related to, and are constrained by, one another

Prior research proposed that trade-off awareness creates unfavorable feelings (p. 400; Luce et al., 1999) – But: the authors show that the TOT feature creates positive emotions!

– Reasons: additional content is conveyed (i.e., relationship among product attribute values) and the interactive presentation

Contribution to task complexity literature (in particular coordinative complexity as one dimension of task complexity) by analyzing how the different number of revealed trade-off relationships influencesn users‘ evaluations

Both enjoyment and product diagnosticity improve perceived decision quality without increasing perceived decision effort

In addition, some practical contributions are derived

60

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Analysis of Links

The motivation identifies a research gap which is directly addressed by the

research objectives

Three theory bases are referenced and integrated in order to develop and

propose a model that adresses the three research objectives

To investigate the proposed model, the authors select a confirmatory research

design. In particular, they select a combination of an experiment and a survey.

The impact of the TOT feature is tested using a between-subjects experiment

design. Further effects are tested using a survey design.

The contributions section directly builds on the analysis of the experiment (and

the survey)

Furthermore, the contributions link back to the motivation by answering the three

identified research objectives

61

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

GCA Experiment: Summary of Analysis

Although a research question is not explicitly stated, the

reader can easily follow the authors‘ work

GCA elements are addressed and links between them are

straightforward

Quality criteria for experiments (e.g., manipulation check)

and surveys (e.g., item loadings) are reported

9 of 10 hypotheses are confirmed and the rejected

hypotheses intuitevely seems to be true in a real setting, too

Limitations are outlined (e.g., students as subjects, little or

no experience with the RA, laptops is a very customizable

product)

Overall, the inferences drawn appear to be valid

62

Brief summary

of article

Motivation

Research Question

Research Objective

Theory Base

Research Design

Contribution

Analysis of Links

Summary of

Analysis


Research

Agenda

Agenda


2 Surveys

3 Experiments


5 Summary


7 References


Software Tool Presentation

Live Demo

– Survey PLS regression in software „SmartPLS Version 2“

– Experiment MANOVA in software „R Studio“


Research

Recommended Materials

PLS algorithm and SmartPLS software:

– Hair, J.F., Hult, T.M., Ringle, C.M., Sarstedt, M. 2013. A Primer on Partial Least

Squares Structural Equation Modeling (PLS-SEM), Sage Publications.

– https://www.youtube.com/user/Gaskination/playlists

Statistical programming language R:

– Just search for it using Google and/or YouTube


Research

https://www.youtube.com/user/Gaskination/playlists

Agenda

Agenda


2 Surveys

3 Experiments


5 Summary


7 References


Today‘s Lecture in Review

67

You learned about the foundations of quantitative, survey-based research

You have some advice on study design

You know about the fundamental process of study design and execution

You are familiar with the most important steps for validating your study

You know the basic quality criteria and strategies to ensure them ( more details in supplementary slides!)

You have the basic tools for the discussion of the survey-based papers

You have seen popular software tools for conducting quantitative research


Research

What did we exclude?

Sampling– In IS important: Who are you talking to? Users? Developers? What is your subject‘s expertise? How often

are they using the IS of interest?...

– E.g. surveying SAP employees about ERP software would probably cause a huge error

Scale development– In case you need to examine new items in your thesis, your supervisor will explain you how to do this

we assumed all questions can be taken from previous literature

– Formative versus reflective measures as long as you can take your questions from preivous literature,

this should not bother you too much

Statistical analyses: Regression, PLS/CB-SEM, (M)AN(C)OVA…– There are multiple specialization courses offered at the University that you can take for this

– Your thesis supervisor can help you in choosing an appropriate statistical method


Research

Questions, Comments, Observations69


Research

Homework until Next Week

Write a Golden Circle Analysis (3 pages text) for one of the following papers

and be able to present and discuss your analysis:

–Survey:

• Li, X., Po-An Hsieh, J. J., and Rai, A. 2013. „Motivational Differences Across Post-

Acceptance Information System Usage Behaviors: An Investigation in the Business

Intelligence Systems Context,“ Information Systems Research (24:3), pp. 659-682

• Note: supplementary material in additional pdf-file!

–Field experiment:

• Martin, S. L., Liao, H., and Campbell, E. M. 2013. „Directive versus Empowering

Leadership: A Field Experiment Comparing Impacts on Task Proficiency and Proactivity,“

Academy of Management Journal (56:5), pp1372-1395

–Experiment + Survey:

• Sun, H. 2013. „A Longitudinal Study of Herd Behavior in the Adoption and Continued Use

of Technology,“ MIS Quarterly (37:4), pp. 1013-1041


Research

Agenda

Agenda


2 Surveys

3 Experiments


5 Summary


7 References


Quality Criteria: Intro to Quality Criteria

Constructs (theoretical level)

– Constructs are imaginary creations in

our minds

– Definitions or constructs are not

objective, but shared (“inter-

subjective”) agreements

Two forms of constructs

– Unidimensional constructs have a

single underlying dimension

– Multidimensional constructs consist of

two or more underlying dimensions

Unobservable theoretical constructs

are translated into indicators

Indicators are questions that can be

empirically observed and measured

Example: socioeconomic status is

measured by asking for

– Family income

– Education

– Occupation

Can be measured multidimensional

or unidimensional

Definition Conceptualization Mental process translating imprecise

concepts into precise definitions

Understand and define what is included

and excluded in a concept

Definition Operationalization Process of developing indicators to

measure abstract constructs

Is based on conceptualization

72

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity

Quality Criteria: Intro to Quality Criteria

Three major types of validity in quantitative research (Cook and Campbell 1979; Shadish et al. 2002):

– Design validity

– Measurement validity

– Inferential validity

Design validity refers to internal validity (causality and control for alternative explanations) and external validity (generalizability)– See previous slides

– E.g., students as participants valid design? potential issues are usually mentioned in the limitation section

Measurement validity estimates how well items measure what they are purported to measure according to their definitions

Inferential validity, also called statistical conclusion validity, refers to the correct application of statistical procedures to find relationships.

Note: This summary of quality criteria focuses on PLS-SEM.– PLS is a prediction-oriented variance-based approach that focuses on endogenous target constructs in the

model and aims at maximizing their explained variance, i.e., their R² value (Hair et al. 2012a).

– PLS has become a quasi-standard (e.g., Bagozzi and Yi 2012; Hair et al. 2012b; Ringle et al. 2012; Shook et al. 2004; Steenkamp and Baumgartner 2000)

– I do not provide a detailed comparison of the two approaches, because it would go beyond the scope of this presentation (e.g., Chin and Newsted 1999; Chin et al. 2003; Marcoulides et al. 2009; Qureshi and Compeau2009) and there is still an ongoing debate about strengths and weaknesses of the two approaches (Goodhue et al. 2012a, 2012b; Marcoulides et al. 2012)


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Two quality goals (Bagozzi and Yi 2012):

– Discriminant validity

– Convergent validity

Discriminant validity refers to the degree to which a construct is more strongly related

to its own indicators than with any other construct (Chin 2010)

– Discriminant validity assures that indicators are assigned to the correct construct

and multiple constructs do not overlap in their definitions

Convergent validity refers to the degree to which a block of items – usually all

indicators of a specific construct – agree (i.e., converge) in their representation of the

construct they are supposed to measure (Chin 2010)

– Convergent validity assures that a set of indicators measures the same construct


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Convergent validity (1/4)

A quasi-standard is reporting factor loadings to show that various indicators are

measuring the same construct.

If loadings would be mixed and have a wide range (e.g., varying from 0.5 to 0.9), this

would raise concern about whether the indicators are a homogenous set that primarily

captures the phenomenon of interest (Chin 2010).

Literature argues that all indicators should be significant, exceed 0.7, and the

difference between indicators measuring the same construct should not exceed 0.2

(Bagozzi and Yi 2012; Chin 2010; Fornell and Larcker 1981).

Similarly, on the construct level, the shared variance of a set of indicators in relation to

their shared variance plus measurement errors, attempts to measure the amount of

variance that a construct extracts from its indicators, so-called average variance

extracted (AVE). It is computed as

where 𝜆𝑖 is the factor loading connecting an indicator to its hypothesized factor and

𝜃𝑖𝑖 is the variance of the error term corresponding to the indicator (Bagozzi and Yi

2012; Fornell and Larcker 1981).75ManTIS FSS 2015 - Quantitative

Research

iii

i

iindicatorfactor

factorAVE

var

var2

2

_

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity



At least 50% variance of items should be accounted for, leading to a minimum AVE of

0.5 (Bagozzi and Yi 2012; Chin 2010; Hair et al. 2014).

Although many researchers have compared the square root of AVE to construct

correlations, you can equivalently compare the AVE to squared correlations among

constructs as this has two advantages (Chin 2010):

– The shared variance is represented in terms of percentage overlap

– Differences are easier to distinguish


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity



Another index measuring a construct’s reliability based on convergent validity of its

indicators, is the composite reliability index 𝜌𝐶 , given by

where 𝜆𝑖𝑗 refers to factor loading i on factor j and 𝜃𝑖𝑖 is the variance of the error term

corresponding to the indicator (Bagozzi and Yi 2012; Werts et al. 1974)

In contrast to Cronbach’s alpha, which is a minimum estimate of reliability, composite

reliability does not assume that all items are equally weighted and thus can be a better

estimate of reliability. Like AVE, 𝜌𝐶 is only applicable for constructs measured with

reflective indicators, too. According to recommendations in literature, 𝜌𝐶 should be at

least greater than 0.6 for new constructs (Hair et al. 2014) and greater than 0.8 for

stable constructs (Fornell and Larcker 1981).


Research

iiij

ij

compositeC

factor

factor

var

var2

2

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity



Furthermore, the average variance an indicator explains is measured through the

communality index and is computed as

where pq is the amount of indicators of the latent variable q and is the

squared factor loading of one indicator of q (Vinzi et al. 2010).

Regarding the entire measurement model, the average communality index is defined

as

where p is the total number of indicators in the model, J is the amount of latent

variables, and pj is the amount of indicators of J (Vinzi et al. 2010).

Note: Although the communality index indicates quality of constructs and the average communality index indicates

the quality of the overall measurement model, communality scores are frequently reported together with quality

criteria of the structural model. The reason for this is, that, based on communality, further indices indicating quality of

the structural model can be calculated and, for the reader, data analysis is be easier to understand if communality

and indices based on communality are reported together.


Research

qp

p

qpq

q

q xcorp

com1

2 ,1

qpqxcor ,2

J

j

jj compp

com1

*1

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Discriminant validity

Recent research recommends to prove that a construct is more correlated with its own

indicators than with the indicators of another construct (Chin 2010). Otherwise there

would be the option that multiple constructs share the same types of indicators and

thus would not be conceptually distinct.

Prior research recommends comparing all constructs’ AVE indices with their respective

squared correlations to other constructs – or equivalently all constructs’ square root of

the AVE indices with their respective correlations to other constructs (Chin 2010;

Fornell and Larcker 1981). If a construct’s AVE score is higher than all squared

correlations to other constructs, the construct is more strongly related to its own

indicators than to the indicators of another construct.

Furthermore, literature recommends comparison of correlations between indicators

and constructs in order to argue for discriminant validity (Chin 2010). That is, loadings

of an indicator to the construct it is supposed to measure (i.e., factor loadings) should

be greater than all loadings of the same indicator to other constructs (i.e., cross

loadings).


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Common method bias (1/2)

Besides convergent and discriminant validity, bias caused by common method

variance (CMV) can be a potential threat, because the exogenous and endogenous

variables are not obtained from different sources (Podsakoff et al. 2003).

The effects of an unmeasured latent methods factor are controlled by including a

common method factor in the PLS model whose indicators included all the principal

constructs’ indicators and should not be significant (Liang et al. 2007; Podsakoff et al.

2003; Richardson et al. 2009; Williams et al. 2003).


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Common method bias (2/2)

Podsakoff et al. (2003) suggest adding a latent method factor to the structural model

which is measured using all indicators. The figure below (adopted from Liang et al.

2007) shows an example with the exogenous variable A and the endogenous variable

B, indicators a1, a2, b1, and b2, measurement errors 𝑒1𝑎, 𝑒2

𝑎, 𝑒1𝑏 and 𝑒2

𝑏, and factor

loadings 𝜆1𝑎, 𝜆2

𝑎, 𝜆1𝑏 and 𝜆2

𝑏.


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Summary of quality criteria for the measurement model


Research



single indicator

High average factor loadings: λ > 0.7

Narrow range of factor loadings: λmax - λmin < 0.2


single construct

High average variance extracted: AVE > 0.5

High composite reliability: ρc > 0.8

High communality index of the construct

Convergent validity of the

measurement modelHigh average communality index


single indicatorEach item loading is greater than all cross-loadings


single construct

A construct’s AVE is greater than the squared construct’s

correlation with any other construct

No common method bias

Substantive factor loadings are greater than method factor

loadings

Method factor loadings are not significant while

substantive factor loadings are significant

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Estimation of standardized path estimates (PLS)

The distribution-free PLS approach estimates standardized path estimates based on

shared variances of the associated constructs

The significance of these estimates are typically assessed using a nonparametric

bootstrapping algorithm (Chin 1998; Chin 2010)

This algorithm is based on n samples with m cases each (Efron and Tibshirani 1993).

– First, for each case all indicators are replaced with a value from their confidence

intervals

– Then, based on m values per indicator, a value for the sample is computed

– This procedure continues until n samples are calculated

The accuracy of the bootstrapping algorithm increases with the amount of cases and

samples on which it is based. Literature recommends using default software properties

for the amount of samples and cases when performing bootstrapping analyzes,

because then research results would be comparable (Temme et al. 2010).

For instance, the bootstrapping algorithm implemented in the software tool

“SmartPLS” estimates (per default) the significance of standardized paths based on

200 samples with 100 cases each


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Interaction effects

Note: Broad IS research has predominantly employed multiple regression- and

ANOVA-based analytic techniques to investigate interaction terms

– You do not have to use PLS for testing interaction effects!

Recent literature suggests to use the product-indicator approach in conjunction with

PLS as described by Chin et al. (2003)

– Advantages:

• This approach requires fewer indicators per construct and a smaller sample

size to find true interaction scores

• Furthermore, it is able to handle measurement error, produce consistent

results, and has a smaller tendency to underestimate paths coefficients

– Disadvantage:

• However, it has a slight tendency to overestimate factor loadings


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity


Predictive Power (1/2)

Besides strengths of associations between various constructs, predictive power of the

structural model needs to be quantified.

Commonly predictive power is reported through R² values of the endogenous

constructs. Falk and Miller (1992) recommend that R² values should be greater than

0.1. Hair et al. (2014) recommend that R² values should be greater than 0.2.

A change in the R² values can further be explored to see whether a particular variable

has a significant effect on another particular variable (Chin 2010). Specifically, the

effect size 𝑓2 should be calculated:

where 𝑅𝑖𝑛𝑐𝑙𝑢𝑑𝑒𝑑2 and 𝑅𝑒𝑥𝑐𝑙𝑢𝑑𝑒𝑑

2 are the R² values provided on the dependent latent

variable when the predicting latent variable is used or omitted in the structural equation

respectively.

According to Cohen (1988), an effect size 𝑓2 of 0.02, 0.15, and 0.35 can be

interpreted as a small, medium, or large impact.


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

2

222

1 included

excludedincluded

R

RRf

Design validity


Predictive Power (2/2)

Besides the R² scores, the redundancy index attempts to measure the quality of the

structural model for an endogenous construct, too (Tenenhaus et al. 2005).

While the R² scores only consider relationships predicting one endogenous construct,

the redundancy index regards the entire structural model (Vinzi et al. 2010).

Furthermore, the redundancy index combines (a part of) the quality of the

measurement model (i.e., communality index) with (a part of) the quality of the

structural model (i.e., R² values):

Likewise, the quality of the overall structural model is expressed by the average

redundancy (Vinzi et al. 2010) computed as

where J is the total number of endogenous latent constructs in the model (Vinzi et al.

2010).


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

2

jjj Rcomred

J

j

jredJ

red1

1

Design validity


Overall (i.e., measurement model + structural model) fit criterion

Furthermore, a global criterion of goodness of fit (GoF) which takes into account the

model performance in both the measurement and the structural model can be

computed

GoF provides a single measure for the overall prediction performance of the model

(Tenenhaus et al. 2005; Vinzi et al. 2010)

GoF is computed as the geometric mean of the average communality and the average

R² value:

Since PLS does not optimize any global function, there is no index that can provide

the user with a global validation of the model (as it is instead the case with 𝜒² [Chi

Square] and related measures; Tenenhaus et al. 2005)

However, the GoF index represents an operational solution to this problem as it may

be meant as an index for validating the PLS model globally (Duarte and Raposo 2010)


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

2RcomGoF Design validity


Multicollinearity (1/3)

While the indices introduced on the previous slides favor strong correlations between

independent and dependent variables, very strong correlations between several

independent variables (commonly known as multicollinearity) are undesired, because

for each regression coefficient there would be an infinite number of combinations of

coefficients that would work equally well thus making it impossible to obtain unique

estimates of the regression coefficients (Field et al. 2012)

– In other words, if there are two predictors that are perfectly correlated, then the

regression coefficients for each variable would be interchangeable

– This could lead to reduced statistical power, untrustworthy regression coefficients,

high sensitivity to small changes in the data, and difficulties to assess the individual

importance of independent variables (Field et al. 2012)

Consequence: correlations between independent variables should be smaller than 0.8

(Field et al. 2012).


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity



Multicollinearity can be detected through the tolerance and variance inflation factor

indices.

Tolerance is the proportion of variance in an independent variable which is not

predicted by the other independent variables (Clark-Carter 2010).

In order to calculate a certain independent variable’s tolerance, that variable is treated

as dependent variable with all other independent variables as predictors. The obtained

R² is then used to determine the variable’s tolerance index:

Similarly, the variance inflation factor (VIF) is computed as


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

21 Rtolerance

21

11

RtoleranceVIF

Design validity



Recent literature argues that tolerance should be greater than 0.1, meaning that at

least 10% of an independent variable’s variance should not be explained by other

independent variables yet (Clark-Carter 2010; Meyers et al. 2006)

Equivalently, VIF should be smaller than 10 (Stevens 2002).

However, O’Brien (2007) argues that the stability of estimated coefficients can be

influenced by other factors. Hence, the variance of the regression coefficients would

be reduced and VIF values of 40 or more could still be acceptable.


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity



Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Summary of quality criteria for the structural model (PLS regression analysis)


Direct effectHigh standardized path estimates

Bootstrap algorithm and t-test

Moderating effect ANOVA and F-test or product-indicator approach in conjunction with PLS

Predicting power

High variance explained (R² > 0.2)

High effect size 𝑓²: 0.02 small effect; 0.15 medium effect; 0.35 large effect

High redundancy

Global quality of structural

modelHigh goodness of fit

No multicollinearity

No perfect correlation between independent variables: Standardized path

estimates < 0.8

High tolerance of independent variables: tolerance > 0.1

Small variance inflation factor of independent variables: VIF < 10

Design validity

Quality Criteria: Design Validity

92

• No responses due to a systematic reason

• E.g., dissatisfied customers tend to be more vocalNon-response bias

• Parts of the population are excluded

• E.g., online surveys exclude people without webSampling bias

• Tendency to portray oneself socially desirable

• E.g., “Have you ever downloaded illegal music?”Social desirability bias

• Participants might not remember certain events

• E.g., “For which tasks have you used your personal computer ten years ago?”

Recall bias

• Variables measured with an identical method, and

• Variables measured at the same timeCommon method bias

Source: Bhattacherjee (2012)


Research

Intro to

Quality Criteria

Measurement

model validity

Structural

model validity

Design validity

Agenda

Agenda


2 Surveys

3 Experiments


5 Summary


7 References


References (1/4)

94

Bagozzi, R. P., and Yi, Y. 2012. “Specification, Evaluation, and Interpretation of Structural Equation Models,” Journal of the Academy of Marketing Science 40(8), pp. 8-34.

Bhattacherjee, A. 2012. Social Science Research: Principles, Methods, and Practices, (2. ed.). Tampa, FL, USA: Open Access Textbooks.

Chin, W. W. 1998. “The Partial Least Squares Approach to Structural Equation Modeling,” in Modern Methods for Business Research, G. A. Marcoulides (ed.), Mahwah, NJ:

pp. 295-336.

Chin, W. W., and Newsted, P. R. 1999. “Structural Equation Modeling Analysis with Small Samples Using Partial Least Squares,” in Statistical Strategies for Small Sample

Research, R. Hoyle (ed.), Thousand Oaks, CA: Sage Publications, pp. 307-341.

Chin, W. W., Marcolin, B. L., and Newsted, P. R. 2003. “A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte

Carlo Simulation Study and an Electronic-Mail Emotion / Adoption Study,” Information Systems Research (14:2), pp. 189-217.

Chin, W. W. 2010. “How to Write Up and Report PLS Analyses,” in Handbook of Partial Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J.

Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 655-690.

Clark-Carter, D. 2010. Quantitative Psychological Research. The Complete Student’s Companion (3rd ed.), Hove, UK: Psychology Press.

Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.), Hillsdale, NJ: Erlbaum.

Cook, T. D., and Campbell, D. T. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings, Boston, MA: Houghton Mifflin Company.

Cronbach, L.J., 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16 (3), 297–334.

Denzin, N. K., and Lincoln, Y. S. 2011. The SAGE Handbook of Qualitative Research, (4. ed.), Thousand Oaks, CA, USA: Sage Publications.

Duarte, P. A. O. and Raposo, M. L. B. 2010. "A PLS Model to Study Brand Preference: An Application to the Mobile Phone Market," in Handbook of Partial Least Squares.

Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 449-485.

Efron, B., and Tibshirani, R. J. 1993. An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability, New York, NY: Chapman & Hall.

Falk, R. F. and Miller, N. B. 1992: A Primer for Soft Modeling, Ohio: The University of Akron Press.

Field, A., Miles, J., and Field, Z. 2012. Discovering Statistics Using R, London, UK: Sage Publications.

Fornell, C., and Larcker D. F. 1981. “Evaluating Structural Equation Models with Unobserved Variables and Measurement Error,” Journal of Marketing Research (18), pp. 39-

50.

Goodhue, D. L., Lewis, W., and Thompson, R. 2012a. “Comparing PLS to Regression and LISREL: A Response to Marcoulides, Chin, and Saunders,” MIS Quarterly (35:3),

pp. 703-716.

Goodhue, D. L., Lewis, W., and Thompson, R. 2012b. “Does PLS have Advantages for Small Sample Size or Non-Normal Data?” MIS Quarterly (36:3), pp. 981-1001.


Research

References (2/4)

95

Goodhue, D., Lewis, W., and Thompson, R. 2006. "PLS, Small Sample Size, and Statistical Power in MIS Research," 39. Annual Hawaii International Conference on System

Sciences (HICSS 2006), Kauai, HI, USA, pp. 202b-202b.

Guba, E.G., and Lincoln, Y.S. 1994. "Competing Paradigms in Qualitative Research," in: Handbook of Qualitative Research, N.K. Denzin and Y.S. Lincoln (eds.). Thousand

Oaks, CA, USA: Sage, pp. 105-117.

Hair, J. F., Ringle, C. M., and Sarstedt, M. 2012a. “Partial Least Squares: The Better Approach to Structural Equation Modeling?” Long Range Planning (45), pp. 312-319.

Hair, J. F., Sarstedt, M., Ringle, C. M., and Mena, J. A. 2012b. “An Assessment of the Use of Partial Least Squares Structural Equation Modeling in Marketing Research,”

Journal of the Academy of Marketing Science (40:3), pp. 414-433.

Hair, J. F., Hult, T. M., Ringle, C. M., Sarstedt, M. 2013. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM): Sage Publications.

Jiang, Z. J., Heng, C. S., and Choi, B. C. F. 2013. „Privacy Concerns and Privacy-Protective Behavior in Synchronous Online Social Interactions,“ Information Systems

Research (24:3), pp. 579-595.

Lewis, B. R., Templeton, G. F., and Byrd, T. A. 2005. “A Methodology for Construct Development in MIS Research,” European Journal of Information Systems (14:4), pp. 388-

400.

Li, X., Po-An Hsieh, J. J., and Rai, A. 2013. „Motivational Differences Across Post-Acceptance Information System Usage Behaviors: An Investigation in the Business

Intelligence Systems Context,“ Information Systems Research (24:3), pp. 659-682.

Liang, H., Saraf, N., Hu, Q., and Xue, Y. 2007. “Assimilation of Enterprise Systems: The Effect of Institutional Pressures and the Mediating Role of Top Management,” MIS

Quarterly (31:1), pp. 59-87.

MacKenzie, S.B., Podsakoff, P.M., and Podsakoff, N.P. 2011. "Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and

Existing Techniques," MIS Quarterly (35:2), pp. 293-A295.

Marcoulides, G. A., Chin, W. W., and Saunders, C. 2012. “When Imprecise Statistical Statements become Problematic: A Response to Goodhue, Lewis, and Thompson,” MIS

Quarterly (36:3), pp. 717-728.

Marcoulides, G. A., Chin, W. W., and Saunders, C. 2009. “A Critical Look at Partial Least Squares Modeling,” MIS Quarterly (33:1), pp. 171-175.

Marcoulides, G.A., and Saunders, C. 2006. "PLS: A Silver Bullet?," MIS Quarterly (30:2), pp. iii-ix.

Martin, S. L., Liao, H., and Campbell, E. M. 2013. „Directive versus Empowering Leadership: A Field Experiment Comparing Impacts on Task Proficiency and Proactivity,“

Academy of Management Journal (56:5), pp1372-1395.

Meyers, L. S., Gamst, G., and Guarino, A. J. 2006. Applied Multivariate Research. Design and Interpretation, Thousand Oaks, CA: Sage Publications.

Monette, D. R., Sullivan, T. J., and DeJong, C. R 2010. Applied Social Research. A Tool for the Human Services, (8. ed.), Belmont, CA, USA: Cengage Learning.

Myers, M. D. 2009. Qualitative Research in Business & Management, Illustrated edition: Sage Publications.


Research

References (3/4)

96

Neuman, W.L. 2000. Social Research Methods: Quantitative and Qualitative Approaches, (4. ed.). Boston, MA, USA: Allyn and Bacon.

O’Brien, R. M. 2007. “A Caution Regarding Rules of Thumb for Variance Inflation Factors,” Quality and Quantity (41), pp. 673-690.

Pagano, R. R. 2010. Understanding Statistics in the Behavioral Sciences, (10. ed.), Belmont, CA, USA: Cengage Learning.

Petter, S., Straub, D. W., and Rai, A. 2007. “Specifying formative Constructs in Information Systems Research,” MIS Quarterly (31:4), pp. 623-656.

Podsakoff, P. M., MacKenzie, S. B., Lee, Y.-J., and Podsakoff, N. P. 2003. “Common Method Biases in Behavioral Research: A Critical Review of the Literature and

Recommended Remedies,” Journal of Applied Psychology (88:5), pp. 879-903.

Qureshi, I., and Compeau, D. 2009. “Assessing Between-Group Differences in Information Systems Research: A Comparison of Covariance- and Component-based SEM,”

MIS Quarterly (33:1), pp. 197-214.

Richardson, H. A., Simmering, M. J., and Sturman, M. C. 2009. “A Tale of Three Perspectives: Examining Post Hoc Statistical Techniques for Detection and Correction of

Common Method Variance,” Organizational Research Methods (12:4), pp. 762-800.

Ringle, C. M., Sarstedt, M, and Straub, D. W. 2012. “A Critical Look at the Use of PLS-SEM in MIS Quarterly,” MIS Quarterly (36:1), iii-xiv.

Shadish, W. R., Cook, T. D., and Campbell, D. T. 2002. Experimental and Quasi-experimental Designs for Generalized Causal Inference, Boston, MA: Houghton-Mifflin.

Shook, C.L., Ketchen, D.J., Hult, T., and Kacmar, K.M. 2004. “An assessment of the use of structural equation modeling in strategic management research,” Strategic

Management Journal, 25(4), pp. 397-404.

Sun, H. 2013. „A Longitudinal Study of Herd Behavior in the Adoption and Continued Use of Technology,“ MIS Quarterly (37:4), pp. 1013-1041.

Steenkamp, J.-B., and Baumgartner, H. 2000. “On the Use of Structural Equation Models for Marketing and Modeling,” International Journal of Research in Marketing (17:2/3),

pp. 195-202.

Stevens, J. P. 2002. Applied Multivariate Statistics for the Social Sciences (4th ed.), Mahwah, NJ: Earlbaum.

Straub, D.W., Gefen, D., and Boudreau, M.-C. 2005. "Quantitative Research," in: Research in Information Systems: A Handbook for Research Supervisors and Their Students,

D. Avison and J. Pries-Heje (eds.). Amsterdam, The Netherlands: Elsevier, pp. 221-238.

Temme, D., Kreis, H., and Hildebrandt, L. 2010. “A Comparison of Current PLS Path Modeling Software: Features, Ease-of-Use, and Performance,” in Handbook of Partial

Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 737-756.

Tenenhaus, M., Vinzi, V. E., Chatelin, Y.-M., and Lauro, C. 2005. “PLS Path Modeling,” Computational Statistics & Data Analysis (48), pp. 159-205.

Venkatesh, V., Brown, S. A., Bala, H. 2013. Bridging the Qualitative-Quantitative Divide: Guidelines for Conducting Mixed Methods Research in Information Systems,“ MIS

Quarterly (37:1), pp. 21-54.


Research

References (4/4)

97

Vinzi, V. E., Trinchera, L., and Amato, S. 2010. “PLS Path Modeling: From Foundations to Recent Developments and Open Issues for Model Assessment and Improvement,” in

Handbook of Partial Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 47-82.

Werts, C. E., Linn, R. L., and Joereskog, K. G. 1974. “Intraclass Reliability Estimates: Testing Structural Assumptions,” Educational and Psychological Measurement (34), pp.

25-33.

Williams, L. J., Edwards, J. R., and Vandenberg, R. J. 2003. “Recent Advances in Causal Modeling Methods in Organizational and Management Research,” Journal of

Management (29:6), pp. 903-936.

Williams, L.J., Vandenberg, R.J., and Edwards, J.R. 2009. "Structural Equation Modeling in Management Research: A Guide for Improved Analysis," The Academy of

Management Annals (3:1), January, pp 543-604.

Xu, J. D., Benbasat, I., and Cenfetelli, R. T. 2014. „The Nature and Consequences of Trade-Off Transparency in the Context of Recommendation Agents,“ MIS Quarterly

(38:2), pp. 379-406.


Research

Contact

Martin KretzerResearch Assistant

Consultation hour: per request

E-Mail: [email protected]

ManTIS 2015 - Overview and

Registration98

mailto:[email protected]

mailto:[email protected]