Effects of unequal indicator intercepts on manifest composite differences

Holger Steinmetz and Peter Schmidt

University of Giessen / Germany

Introduction

Importance of analyses of mean differences

For instance:- gender differences on wellbeing, self-esteem, abilities, behavior- differences between leaders and non-leaders on intelligence and personality traits

- differences between cultural populations on psychological competencies, values, wellbeing

Usual procedure: t-test or ANOVA with manifest composite scores

Latent variables vs. manifest variables

Manifest mean = indicator intercept + factor loading * latent mean

→ Will unequal intercepts lead to wrong conclusions regarding composite differences?

Intercepts and latent means

eXBBY 10

X Y

Y

X

B1

B0

Intercepts and latent means

iiiix

xi

i

i

x1

x4

x2

x3

Intercepts and latent means

)()()( iiii EExE

xi

i

i

x1

x4

x2

x3

Intercepts and latent means

x1

x4

x2

x3

iiixM )(

xi

i

i

M(xi)

Intercepts and latent means

x1

x4

x2

x3

iiixM )(

xi

i

i

M(xi)

Group differences in intercepts and factor loadings

xi

M(xi)M(xi)

M(xi)

x1

x4

x2

x3

x1

x4

x2

x3

Group A Group B

Group differences in intercepts and factor loadings

xi

M(xi)M(xi)

M(xi)

x1

x4

x2

x3

x1

x4

x2

x3

Group A Group B

Group differences in intercepts and factor loadings

xi

M(xi)M(xi)

M(xi)

x1

x4

x2

x3

x1

x4

x2

x3

Group A Group B

Meaning of (unequal) intercepts

Associated terms used in the literature- Item bias- Differential item functioning- Measurement/factorial invariance ("strong factorial invariance", "scalar invariance")

Meaning- Response style (acquiescence, leniency, severity)- Response sets (e.g., social desirability)- Connotations of items- Item specific difficulty

The study

Partial invariance

Research question: Is partial invariance enough for composite mean difference testing?

- Pseudo-differences

- Compensation effects

Procedure (Mplus):

- Step 1: Specification of two-group population models with latent mean and intercept differences; 1000 replications,

raw data saved

- Step 2: Creation of a composite score

- Step 3: Analysis of composite differences

- Step 4: Aggregation (-> sampling distribution)

The study

Design (population model):- Two groups- One latent variable- 4 vs. 6 indicators- All intercepts equal vs. one vs. two intercepts unequal in varying directions (+.30 vs. -.30)

- Latent mean difference: 0 vs. .30- Loadings kept equal with ‘s = .80; latent variance = 1- N = 2 x 100 vs. 2 x 300- Latent models as comparison standard for each condition

Dependent variables- Average composite mean difference - Percent of significant composite differences („% sig“)

Full scalar invariance(Latent mean difference = .30)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Avg. composite difference

%sig

N = 2 x 100 N = 2 x 300

4 Indicators 6 Indicators 4 Indicators 6 Indicators

Pseudo-Differences(Latent mean difference = 0; unequal intercept(s)

4 Ind. 6 Ind.

N = 2 x 300 N = 2 x 100 N = 2 x 300

2 Intercepts unequal (.30)

4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind.

N = 2 x 100

1 Intercept unequal (.30)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

Avg. composite difference

%sig

Pseudo-Differences(Latent mean difference = 0; unequal intercept(s)

4 Ind. 6 Ind.

N = 2 x 300 N = 2 x 100 N = 2 x 300

2 Intercepts unequal (.30)

4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind.

N = 2 x 100

1 Intercept unequal (.30)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

Avg. composite difference

%sig

Compensation effects(Latent mean difference = .30; negative intercept

difference)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

Avg. Composite difference

%sig

4 Ind. 6 Ind.

N = 2 x 300 N = 2 x 100 N = 2 x 300

2 Intercepts unequal (-.30)

4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind.

N = 2 x 100

1 Intercept unequal (-.30)

Compensation effects(Latent mean difference = .30; negative intercept

difference)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

Avg. Composite difference

%sig

4 Ind. 6 Ind.

N = 2 x 300 N = 2 x 100 N = 2 x 300

2 Intercepts unequal (-.30)

4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind.

N = 2 x 100

1 Intercept unequal (-.30)

Summary

Full latent variable models have more power than composite analyses

Pseudo-differences- Even one unequal intercept increases the risk to find spurious composite differences

- High sample size increases risk- Number of indicators reduces the risk – but not substantially

Componensation effects- Even one unequal intercept reduces the size of the composite difference to 50%

- In small samples little chance to find a significant composite difference (power = .25 - .40)

- Two unequal intercepts drastically reduce the composite difference: The power in the „best“ condition (2x300, 6 Ind.) is only .50

Conclusons

Most comparisons of means rely on traditional composite difference analysis

These methods make assumptions that are unrealistic (i.e., full invariance of intercepts)

Even minor violations of these assumptions increase the risk of drawing wrong conclusions

Advantages of SEM:- Assumptions can be tested- Partial invariance implies no danger- Greater power even in small samples

Thank you very much!