Mosteller & Tukey (1977). Data Analysis and
Regression.
Slide 3
Gerry Altmann Encouraging linguists to use linear mixed-
effects models is like giving shotguns to toddlers. (see Barr et
al., 2013)
Slide 4
A world of subjectivity Sarah Depaoli IF YOU BEAT THE DATA, AT
SOME TIME IT WILL SPEAK
Slide 5
A world of subjectivity Sarah Depaoli and then you publish and
get tenure.
Slide 6
LMM response ~ intercept + slope * fixed effect + error
distinguish between test and control variables
Slide 7
Test vs. Control Variable Example test variablecontrol variable
Null Model
Slide 8
Test vs. Control Variable Example test variablecontrol variable
Null Model
Slide 9
Test vs. Control Variable Example Critical Effect Control 1
Control 2 Random Effects Response ~ BLACK BOX BLACK BOX
Slide 10
Test vs. Control Variable Example Critical Effect Control 2
Random Effects Response ~ BLACK BOX BLACK BOX Control 3
Slide 11
Test vs. Control Variable Example Critical Effect Control 2
Random Effects Response ~ Control 3
Slide 12
Trade-off #1 Model Simplicity Model Fit
Slide 13
Trade-off #2 Data- driven Theory- driven Exploratory End
Confirmatory End Harald Baayen
Slide 14
Trade-off #2 Data- driven Theory- driven Exploratory End
Confirmatory End Roger Mundry (and many others)
Slide 15
How much do you allow the data to suggest new hypotheses? How
much do you depend on a priori theory? Trade-off #2 Big
Question:
Slide 16
Approach 1: more data-driven Approach 1: more data-driven
Approach 2: more theory- driven Approach 2: more theory- driven
e.g., test whether random slopes are needed (maybe not advisable)
e.g., test whether interaction for sth. is necessary or not (o.k.
if the interaction is a control variable) e.g., test whether sth.
requires a non-linear or a linear effect (maybe o.k.)
Slide 17
THINGS TO WORRY ABOUT: Taken to the extreme, this approach has
a very high likelihood of finding any significant result The model
selection process is less transparent to outsiders (or, you have to
write a LONG LONG stats section) Approach 1: more data-driven
Approach 1: more data-driven Approach 2: more theory- driven
Approach 2: more theory- driven
Slide 18
Approach 1: more data-driven Approach 1: more data-driven
Approach 2: more theory- driven Approach 2: more theory- driven
ADVANTAGES: You dont miss important things in your data Your model
might thus be more accurate and more true to the data
Slide 19
Approach 1: more data-driven Approach 1: more data-driven
Approach 2: more theory- driven Approach 2: more theory- driven You
formulate your model before you look at the data The components of
your model are guided by: Theory + Published Results General
world-knowledge Research experience Taken to the extreme, you cant
even make a plot before you formulate your model
Slide 20
Approach 1: more data-driven Approach 1: more data-driven
Approach 2: more theory- driven Approach 2: more theory- driven
ADVANTAGES: It forces you to think a lot Its fun! It gives you a
lot of responsibility, as a scientist Your estimates are going to
be more conservative
Slide 21
Approach 1: more data-driven Approach 1: more data-driven
Approach 2: more theory- driven Approach 2: more theory- driven
Think about model (before you conduct your experiment) Build model,
evaluate the models assumptions Build model that better fits the
assumptions Test whether control variables interact with test
variable, or whether they are needed
Slide 22
Dialogue with your model You need to know that theres multiple
responses per subject and item! People might speed up or slow down
throughout an experiment. You need to know that each item was
repeated two times! Token Researcher ;-)
Slide 23
Keep in mind: You have to resolve non-independencies Your
random effects structure should be maximal with respect to your
experimental design
Slide 24
Protecting your research from yourself: Whatever you do, your
model decision should not be based on the significance of your
effect
Slide 25
(JEPS Bulletin)
Slide 26
Important principle CONFIRM FIRST EXPLORE SECOND John McArdle
McArdle, J. J. (2011). Some ethical issues in factor analysis. In
A.T. Panter & S. K. Sterba (Eds.), Handbook of Ethics in
Quantitative Methodology (pp. 313-339). New York, NY: Routledge.
McArdle (2011: 335)
Slide 27
The write-up
Slide 28
Important principle BE HONEST NOT PURE John McArdle
Slide 29
Cool guidelines United Nations Economic Commission for Europe
(2009a). Making Data Meaningful Part 1: A guide to writing stories
about numbers. New York and Geneva: United Nations. United Nations
Economic Commission for Europe (2009b). Making Data Meaningful Part
2: A guide to presenting statistics. New York and Geneva: United
Nations.
Slide 30
We tested a linear mixed effects model with subjects and items
as random effects.
Slide 31
The write-up should reflect (as adequately as possible) the
details of your model and your model selection procedure =
Reproducible Research
Slide 32
Rule of thumb: One needs to provide sufficient information for
the reader to be able to recreate the analyses. Barr et al. (2013)
Ask yourself: With the information that I provided, could I,
myself, replicate the analysis?
Slide 33
How to write up (1) "Phenomenon-oriented write-up" (2) Appendix
/ Supplementary Materials
Slide 34
We used generalized linear mixed models to test the effect of
Gender and Politeness on pitch. Subjects and items were random
effects (random intercepts) (Baayen, Davidson & Bates, 2008),
with random slopes for subjects and items for the effect Politeness
(Barr, Levy, Scheepers & Tily, 2013). We also included a Gender
* Politeness interaction into the model and if this interaction was
not significant, only included the main effects. /// Q-Q plots and
plots of residuals against fitted values revealed no obvious
deviations from normality and homoskedasticity. We report p-values
based on Likelihood Ratio Tests of the model with the main fixed
effect in question (Politeness) against the model without the main
fixed effect (null model, including Gender). Example #1
Slide 35
We used generalized linear mixed models to test the association
between voice onset time and pitch. The fixed effects quantify the
effect of VOT on politeness, as well as the effect of place of
articulation, vowel type and gender on politeness. The random
effects quantify the by-subject and by- item variability in pitch
(random intercepts), as well as the variation of the effect of VOT
on pitch for subjects and items (random slopes). Example #2:
"Phenomenon-oriented"
Slide 36
Visual inspection of residual plots revealed no obvious
deviation from normality and homoskedasticity of errors. We checked
plots of residuals against fitted values and found no indication
that the normality and homoskedasticity assumption were violated.
indicated a problem with We therefore log-transformed the data.
Mentioning assumptions
Slide 37
Results o Provide results of likelihood ratio test (i.e.,
significance etc.) o Provide estimates and standard errors in the
metric of the model o For poisson and logistic regression,
additionally provide some exemplary back- transformed values (dont
back-transform the standard errors)